Built With Our Members, Not For Them: How We Launched Lloyd AI

When we set out to bring AI to the Braintrust, we made one deliberate choice that shaped everything that followed: we would not build Lloyd in a vacuum and drop it on members. We would build it with their input from the very beginning.

That decision led to two distinct programs before public launch – a 6-week private design partner program with our most AI-forward members, followed by a 3-week private beta with a broader cross-section of engaged legal leaders. Together, they changed the product in ways we couldn't have anticipated sitting in a room by ourselves.

Why We Did This Before Launching

Trust is the foundation of The L Suite community. Members share hiring dilemmas, founder conflicts, board tensions, and sensitive contract questions with their peers – conversations they'd never have on a general-purpose AI tool. Bringing AI into that space required more than good technology; it required earning the right to be useful.

So before any broad rollout, we pressure-tested Lloyd with a cross-section of members: GCs at high-growth tech companies, Deputy GCs, Legal Ops leaders, venture fund counsel. People who use AI regularly and know when it's good versus when it's generic.

The goal wasn't theoretical accuracy. It was real-world usefulness, honest friction, and a genuine commitment to building with – not for.

How the Programs Were Structured

The design partner program came first: 50 of our most AI-forward members currently in seat. We knew this cohort would push hard, find the edges, and tell us the truth. From there, we opened a broader private beta to 500 members across peer networks, mimicking a community-wide release.

Each week, the design partners had a focused theme to explore – a human question like Does Lloyd think like us?, Would I trust this with real work? And what happens when things get messy? – paired with a specific prompt and a reflection that members shared back via Slack and a weekly survey.

We also built in a recurring "Wreck-It Wednesday," explicitly inviting members to surface what wasn't working and gamifying the hunt for bugs. Negative feedback, we've learned, is the most valuable kind, and you have to design for it.

5 Things We Heard (and What We Changed)

Across both programs, five themes emerged that directly shaped the product.

1. The responses were too long. When members needed a quick answer, Lloyd was giving them an essay. We recalibrated response length across query types – shorter for factual lookups, more structured for complex analysis – and introduced a tighter default format for follow-up prompts.

2. Source citation needed to work better. Members loved the citation sidebar concept but reported broken links, mislabeled citations, and sources that timed out rather than loading. We fixed source validation and improved how citations are labeled and surfaced.

3. Document export was broken. Several members tried downloading Lloyd's output in Word format and received garbled, unreadable text. We resolved the core export bug and improved formatting fidelity for downloadable responses.

4. Edge case prompt handling needed work. Members using complex, multi-part prompts or referencing specific documents occasionally saw Lloyd stall, loop, or surface off-topic context. We tightened edge case handling and improved how Lloyd interprets ambiguous instructions.

5. The Braintrust knowledge layer needed to surface more proactively. Rather than waiting to be asked, Lloyd now does a better job of pulling in peer-generated context – relevant threads, discussions, and templates – even when members don't explicitly prompt for it.

In Their Own Words

"Lloyd gave me practical, peer-informed answers on everything from California remote work compliance to EU regulatory crisis triage. I deliberately tried to get it to make up an answer on a technical DSA question and it refused, which is exactly the kind of intellectual honesty I need from a tool I'm going to rely on for real work. The community knowledge layer is what sets it apart." — Smrithi Mohan, GC @ Awesome

"Lloyd’s community knowledge layer brings years of in-house legal best practices and operational know-how right to your fingertips – everything from detailed community prompts aligned to purpose and risk profile to pressure testing and benchmarking a position, [all] powered by the shared wisdom and thoughtfulness The L Suite is known for." — Sonal Mehta, GC @ BetterCloud 

"Unlike most of the conversational chatbots I've tried, Lloyd is meticulous in its research, and leads with its source material. It's immediately become one of my go-to tools when tackling a novel issue or looking to quickly scale up on a topic or process." — Angela Markle, AGC @ Weedmaps

What Both Programs Taught Us About AI in a Community

The clearest signal across both cohorts: members don't want AI to replace the community. They want AI to help them use the community better.

The most resonant use cases weren't the ones where Lloyd went deep on legal research. They were the ones where Lloyd surfaced what peers had already figured out – and then pointed members to the right person to continue the conversation. That's a different kind of AI tool. It's a social intelligence layer, not a search engine.

We built Lloyd to reflect that distinction. The beta helped us prove it.

What Comes Next

Lloyd is now live for all L Suite members, with improvements from both programs already built in. Coming soon:

• A redesigned document export experience with reliable Word and PDF downloads

• Expanded the redlining capabilities that members have been asking for

• Deeper Braintrust surfacing that proactively connects Lloyd's responses to the most relevant peer discussions, so you spend less time searching and more time acting on what your community already knows.

For every member who tested, challenged, praised, and pushed Lloyd over these past months: this is your product. The version launching now is better because of you.