Building Your First Internal AI Prompt Library

With “slop” on the rise, you might wonder whether now is the time to dive deeper into AI prompting. Most of the large AI vendors offer their own robust prompt libraries, many of which are highly tailored to legal use cases. But developing an internal AI prompt library can be a valuable exercise. 

If your company’s AI policy sets the ground rules for using AI internally, an internal prompt library helps the whole team put the policy into action tactically while improving output and capacity. This article will explore the unexpected benefits of creating a legal-team AI prompt library, as well as practical techniques for building your first library, tracking its success, managing it over time, and overcoming common points of friction.

Unexpected Benefits of Creating a Prompt Library

Once the legal team is using AI regularly, a prompt library is more than just a nice-to-have. Prompt libraries can save time on reinventing the wheel for routine tasks while increasing consistency between team members and producing a shared baseline for quality control. Beyond these advantages, here are some unexpected benefits that an AI prompt library can deliver:

• Improving knowledge transfer and coverage between teammates, which is helpful when a new hire joins or someone is out of office

• Manages the risk of vendor lock-in by making your team more agnostic across providers, while customizing prompts to your company's playbooks and risk posture rather than relying on vendor default options

• Standardizing prompt protocols could reduce potential security risks while improving auditability and mitigating some of AI’s known limitations like bias and hallucinations

Building and Launching Your Prompt Library

Creating a first AI prompt library can be relatively low lift, and then improved from there. Here are three tactics to prototype your prompt library beyond simply asking your team for their favorites (which can work, too!):

• List the artifacts you create most often (e.g., summaries, contract redlines) and draft prompts for each output listed using AI

• Have AI review materials with your default position (e.g., contract playbooks, intake questionnaires) and generate prompt templates

• Export previous prompts from your AI tool and save the most common prompts per task

Next, review the prompts to check they have the correct assumptions on subject matter and output format. Tag them by risk level and specify when human review is required, as well as any specific situations where AI should never be used. Share with the team for a short pilot to find out where the library needs further tweaks. After making these changes, “v1” of the AI library is ready.

If you need inspiration on creating reusable prompts, read The L Suite’s guide to mastering AI prompts, which includes a member-generated “super prompt”. For an example prompt library structure, consider exploring this one from Wharton, which was created by Ethan and Lilach Mollick. The L Suite members also have access to a full library of prompts from the community in the Braintrust platform.

Defining and Tracking Success

Visualizing and then tracking your AI prompt library’s effectiveness is essential for making sure it serves your business. Here are some metrics to evaluate how well your AI prompt library is working, and identifying where to fine-tune if it falls short. Note that assessing the prompt library’s impact can be a great way to partner with legal operations if your company has this function.

• Efficiency: Is the cycle time decreasing for routine work? Is capacity per lawyer increasing? Are questions to subject matter experts becoming more precise?

• Usage and Satisfaction: Is the team actively using the library? Are they having a positive experience with it?

• Knowledge Transfer: Is there better coverage between employees when one goes on leave? Are new hires getting up to speed faster?

Ongoing Maintenance

Keep tending to the prompt library after launch to make sure it stays relevant and matches your business evolution. Here are the first steps to take:

• Version Control: Assign each prompt category an owner. List version numbers and date of last review. Plan to review the prompts at least quarterly.

• Customization: Evaluate or add parameters that give the prompt more context to produce an accurate result (e.g., jurisdiction, product stage, counterparty, customer segment, risk tolerance) as the business changes.

• Additions and Deletions: Decide who will approve new prompts and any criteria for removing stale prompts. Create a submission link for proposed changes.

• Evaluation: Regular use will likely surface issues organically, but test the prompts with edge cases periodically to find new areas of improvement and confidence levels in the output. Check whether the prompts work equally well across vendors and whether human review protocols continue to match your company risk posture.

Managing Common Pitfalls

Here are some techniques to navigate some of the more common pitfalls you might encounter as you launch your first AI prompt library:

• Bandwidth: Triage the most frequently used prompts so they get regular review. Archive prompts that are not being used. Rotate which legal team member is responsible for the library.

• Privilege and Discoverability: Limit access to sensitive prompts. Consider adding language reminding users to use only approved AI tools, redact details, and review the output before sharing. Work with IT on retention controls where possible.

• Default Policy Creation: To prevent AI-generated positions from inadvertently becoming the company’s default, include language in the prompt reminding reviewers to confirm outputs against the company playbook.

Conclusion

An AI prompt library can increase speed and consistency throughout the team, leading to improved knowledge, risk, and cost management, as well as reduced vendor lock-in. Launch a small pilot with prompts for your most frequently completed (lower-risk) tasks, and collect feedback.

As you scale up, include tags on risk level, jurisdiction, and human review. Assign an owner to each prompt category, and create a habit of reviewing and evaluating the prompts to make sure they are meeting your goals in partnership with legal operations. Over time, an AI prompt library can be one of the best investments your company can make in using AI with clarity and precision.