Today we are excited to begin rolling out access to Adept Experiments, to share some of the capabilities we are building for our enterprise use cases on our way to useful general intelligence. Each experiment is a self-contained mini-tool or demo that showcases a part of our underlying tech, and that we would love you to play with, experience, and share feedback about.

The first experiment is a little workflow builder for the web that you can configure with plain language.

Our first Experiment: Workflows

At this early stage, we’ve learned that the most valuable things an AI teammate can help you with are software workflows that are specific to you, your job, or your company. For example, even a common task like adding a lead to a CRM looks very different from company to company.

We believe the foundational skill for an AI teammate is to 1) quickly learn a tedious or complex task from a user and 2) reliably run it. This is the capability we’re demonstrating in our first experiment: Workflows.

A few caveats:

• Because Workflows is an Experiment, it often requires some careful prompting to make it do what you want.
• Some sites work better than others; by working directly with us to enable their workflows, our enterprise customers experience >95% reliability.

Even with these limitations, we’re excited about the scope of fun and useful workflows that are possible. Below are some examples we’ve explored with users to get you started — we’re excited to see what you build!

Delegating repetitive knowledge work tasks to Adept

A corporate recruiter receives numerous emails each day from candidates as they move through the hiring process. With Adept, they can set up a custom workflow that advances the candidate to the next stage and requests their availability with a single click of a button. In this case, the workflow identifies the candidate and moves them to a technical interview round. This can be done in a background tab while the recruiter moves on to other work.

Turning unstructured data into structured data

Accounting managers often handle invoices from contractors over email. Here is a workflow that enables Adept to open attached invoices, extract information such as invoice numbers and total cost, and enter this information into their accounts payable software.

Hopping between tools

The average knowledge worker uses 17 different software tools as part of their job and spends much of their day shuttling data back and forth between them. Adept can already run workflows that bridge different software tools, and since it can understand the screen context from the source tool as it acts in the destination tool, it can do this super efficiently.

For example, a fair chunk of insurance agents’ work involves extracting information from claims emails and inserting the data into forms in other software tools. Here’s how Adept can make that process a lot less painful!

Doing things you don’t know how to do

Sometimes we spend hours searching through documentation, watching tutorials, and asking coworkers how to get something done. With Adept, a novice user can navigate the tool like an expert.

A new store manager may not know how to use every feature on Shopify yet, but they can use Adept (trained by a more seasoned colleague) to easily create new discount codes.

Order dinner!

How Workflows Work

Workflows is powered by ACT-2, a model fine-tuned from the Fuyu family and optimized for UI understanding, knowledge worker data comprehension, and action taking. In some cases, we also use language-specific models when doing simple tasks like text composition. Here are a few highlights:

Workflows see like human eyes

Adept is an end-to-end multimodal AI agent. It uses software just like a person would: it can perceive the screen directly via pixels and act on your computer through coordinates and keystrokes. This makes it infinitely extensible without the need to create hundreds of API integrations, manage user login credentials, etc. Moreover, Adept can take actions across all software, even when a text or HTML representation does not exist or is not sufficient. For example, web apps like Bing Maps use a 2D canvas, which until now has been too difficult to manipulate programmatically.

Workflows are easily teachable

Use cases are as diverse as users are — and each enterprise comes with their own context and tools. For this reason, we’ve built Adept to learn a new workflow in minutes.

In Workflows, tasks are represented by a series of actions like “click” and “type”. In the future, we expect users to be able to instruct Adept at increasingly higher levels of abstraction.

Workflows put you in the driver’s seat

We’ve designed Adept to work hand-in-hand with the user. You can see some of our design principles in action with Workflows. When taking actions, Adept prompts the user for information needed to complete the task. Then it either performs actions one at a time (when creating or testing a workflow) or auto-runs all actions, with each step visible to the user.

We share the community’s concerns about the safety of fully autonomous agents; for that reason, Adept is designed for human-in-the-loop supervision.

What’s next

We expect AI agents to improve tremendously over time, similar to the advances we’ve all seen with language and image foundation models over the last few years. In particular, we’re excited about our research roadmap focused on higher-level planning, improved visual reasoning, incorporating enterprise context, and learning from demonstrations. Stay tuned for more Experiments and research updates as we continue to iterate.

Adept is already improving day-to-day work for people at our first enterprise customers. We’re currently selecting a small number of additional partners for 2024 — if you’re interested in becoming an early enterprise customer, please complete our enterprise inquiry form here.

Start experimenting!

We will onboard new users on a rolling basis. Thanks for joining us on the journey!

Thanks to all who made this possible, including NVIDIA, Microsoft, Oracle, WEKA, Plasmo and our investors and advisors.