Today we're building a copilot agent using computer
use within Microsoft Copilot Studio. This is
one of the newest tools available and using natural
language you tell this tool what to do and it
drives a virtual mouse and keyboard and gets
the task done for you. You have to see it to
believe it. We'll be building this agent step
by step so that you can follow along. My name
is Shervin Shafi and I'm a Co -Pilot Principal
Engineer at Microsoft. Welcome to Collaboration
Simplified. Okay, so let's jump straight into
building this agent. We're inside Copilot Studio
and we're looking at an agent that I actually
built in the past. It was called the HR Miracle
Worker Pro. And if you want to know how it was
built, it was basically in one of my videos in
the past in this Copilot Agents one right over
here. So if you want to see how that was built,
it was demonstrated in this video. But the reason
why I'm showing it to you is because I'm going
to rebuild it this time using computer use. So
the reason I'm rebuilding it is because this
particular agent, HR Miracle Worker Pro, it basically
connects to ServiceNow and it does it using connectors
or API. So you can see under the tool section
here, there's this tool called List Records.
So if I open it up, you can see that basically
it connects to ServiceNow. There's even more
to it than that. This is actually the easy part.
But on the top here, if I click this plus five,
I can go to a section called Topics. And when
I go into Topics, there's a section here called
Booking Vacation that I created. And when I open
that up, there's a flow in here that says, look,
if somebody is asking you to book a vacation,
this is a trigger. Ask them these questions and
ultimately it goes ahead and it connects to this
custom connector. And again, the reason I'm showing
you all this is to kind of show you all the work
that has to go into play to create a connection
to ServiceNow. So this particular connector,
if we go to the designer section of it inside
of Flow, not only does it list the record over
here, there's some work involved there, but for
each transaction, it also updates the record.
And when we look at updating the record here,
it looks at hours that were booked someone's
trying to book vacation we want to see how many
hours they had booked and hours they have left
and there's formulas in there we want to bypass
all this right we don't want to have to do all
this work we just want to tell this computer
use agent click here click there and just do
it without having to do a little bit of low code
if you will now what do we want in terms of steps
this computer use to do. So this is what we wanted
to do. I have a ServiceNow developer login. And
so what I want this computer use to do is, first,
I wanted to go to this website, developer .servicenow
.com. Then I wanted to click on sign in, then
I wanted to sign in with my login of I love computer
use at outlook .com. And then put in my password
of copilot studio rocks and then click sign in
and then once it signs in i wanted to go to this
section here under app engine studio i wanted
to select that and then once it selects that
i wanted to scroll down to this my app section
and click on hr and once it does that i wanted
to close that out and then go here and click
on this hrdb this hr database and then once that's
done i want it to go over here on the top and
i wanted to select preview and then it gets this
table of my employees and basically i have four
employees i have jane pete sammy and bob and
then i can see how many hours they've booked
and how many hours they have left and then i
wanted to update these hours by what is being
requested of it that other agent miracle worker
pro did all that using connectors or apis but
i want computer use to do it by just clicking
and using the keyboard and mouse and getting
to it and updating these fields because ultimately
if i wanted to update jane's profile i wanted
to click on jane i wanted to go to these numbers
and update them and then click update all right
so that's the use case this is what we're going
to build together To build it, we start off in
Copilot Studio and on the top left -hand side,
I'm gonna select create and I'm gonna select
new agent and I'm just gonna select create on
the top right -hand side to get the template
of the agent created. So we'll just give it a
sec here. Now I can go ahead and select edit
over here to edit the agent name and we're gonna
call it computer use. for service now and i'm
going to click on change the icon over here and
then again here change icon i have clippy so
let's use this one let's go ahead and click edit
again and give it a description for the agent
an agent that can book vacation days for employees
okay and then we're going to click save on the
top right hand side here so we got the name We
gave it a picture. We gave it the description.
And it's using the GPT -4 -0 model. And by the
way, if you were curious, if you select settings
for the agent, you might be able to, in your
version, change the models to different ones,
like 4 -1, GPT -5 auto, GPT -5 reasoning, and
so on. So I'm going to stick with the default
at the moment, but just kind of a heads up. And
there's some other things you can change here,
like content moderation. Right now it's set to
high. Maybe you want to set it to low just to
kind of experiment. So there's all these kinds
of things you can check. And also here, sometimes
people are like, well, it's using the knowledge
of the large language models. And then you can
actually turn it off so that general knowledge
isn't used. But we're going to leave it on for
now. So there's some cool settings kind of as
a pro tip if you want to click. on the top right
hand side you get to all those sections okay
now what are we going to do in terms of instructions
well i needed to kind of flow where it asks a
bunch of questions like who are you booking this
vacation for and then i wanted to get to the
computer use so first what we're going to do
is we're going to go to this plus 5 over here
and we're going to create a topic And here I
selected this arrow next to it and I'm going
to add from a description with Copilot. So here
I'm going to call it booking vacation. And I'm
going to create a topic. I'm going to tell it
what I wanted to do. So get the first and last
name of the person looking to book vacation.
as well as how many hours they want to book all
right let's go ahead and create that see what
it does close this out close out this side here
and so basically it says this tool handles queries
like book a vacation plan my holiday and then
the first question this is all auto generated
based on what i just wrote it what is your first
and last name and it says identify the person
as their first and last name over here and what
i want to do is i just actually select it on
the person's name just select it and then i want
to do multiple choice options so i just want
to give it that information so i'm going to select
multiple choice and it says option for the user
if i actually go to my database for service now
these are the names of the people that i had
Jane Ingley, Pete PD. Let's put those people
in. So go back here and for the first option,
we're going to put Jane Ingley. And then click
enter. Plus new option, Pete PD. Don't tell me
how I came up with these names. Click new option,
Sammy Belong. I'm going to put Bob Smith. All
right, so hopefully those are all the right names.
So we're asking for the first and last name.
We're giving options. And then this all got auto
-created. So based on what it equals, like all
of these feed into the next questions. How many
hours do you want to book? It's a number and
the response is saved as hours. The other response
was saved as name. And so I think this is good
for now, right? So I'm just going to go ahead
and on the top right hand side, I'm just going
to click save. So this is a topic. What we're
going to do next is eventually we're going to
pull all this together in the instructions. But
at the moment that I've clicked save, I'm going
to click on overview on the top here. And I'm
going to scroll down to the tool section so we
can use, we can add our computer use. So I go,
I'm going to go ahead and select add a tool.
And this is where you would generally search
for all these connectors, right? So if I searched
for ServiceNow, those different ServiceNow connectors
that we used in the other agent, they all pop
up here, but we're not going to do that. What
we're going to do instead is I'm going to go
ahead and select plus new tool over here. And
then this is where all those tools show up, including
computer use. in the past we actually did a video
on the model context protocol that's actually
a cool one where we connected to DocuSign and
then today we're going to do computer use which
again you can see it's in preview at the moment
and by the way real quick we have some really
good documentation on computer use so if you
go to the Copilot Studio admin guide this is
the documentation for computer use right here
and it was posted maybe about a month ago and
so definitely check that out there's a good video
in here it talks about how to add it to your
agent which we're doing at the moment all kinds
of good stuff in this documentation now back
to the agent we'll just select computer use and
here it says look what are the instructions that
you want to put in for the computer use for example
go to this website add an invoice all this kind
of stuff or do you want to start with a template
you know for invoice processing or for data entry
and if you select invoice processing or data
entry or data extraction it actually puts a sample
set of instructions in here and these websites
are actually live so this is a good way for you
to experiment if you don't have a place to experiment
with so definitely check this out and these are
also in the video and in that link as well in
the documentation now for now i'm just going
to go ahead and erase all this i'm not going
to put any instructions the other thing to note
is that it says look how are you connecting to
this computer that's running these instructions
and i'm using a hosted browser which if you look
for what it is if you click the i and go to learn
more it opens up that site that i showed you
and if you scroll all the way down to a section
in the bottom here it says configure where your
computer use runs. So I'm gonna be using this
hosted browser powered by Windows 365, which
basically allows me to start quickly without
any machine setup. And this hosted browser runs
in a Microsoft managed environment, but it is
not Microsoft Entra. joined to your tenant or
managed by your Intune policies. So this means
that if you're trying to connect to your SharePoint
site or to your OneDrive or a website that's
internal, it's not going to be able to do that.
But if you want to connect to a public site like
I am to this ServiceNow developer site, it can
connect to it, right? There is a way you can
actually connect to your intro connected domain
managed machine, which is right here. And it
tells you how to go about doing that. We're not
going to do that, but the documentation shows
you how to do that if that's what your use case
is for. So for us, we're just going to leave
it as use hosted browser and then go ahead and
select add and configure on the bottom here.
Now we're in this section where we have details
and inputs. And for the name of it, I want to
say this is computer use to service now. okay
description what does it do connects to the service
now developer site that has employee vacation
hours all right now here's where all the instructions
come everything that i showed you we expect this
to do in terms of go here click this do that
this is where we're going to put the instructions
now i've come up with the instructions because
there's a bunch of them so i'm just going to
copy and paste them okay so i just pasted everything
in and let's take a look at these together right
so first we're going to go to the particular
website which is at developer .service .com so
we're telling go there and if you're not logged
in click sign in and then log in with my email
and my password this is not a secure way of doing
it but i'm doing it just because this is a demo
what you're supposed to do officially is you
go to this stored credentials section here and
you add your credentials and this adds them to
your azure key vault in a very secure way and
then you can use them i didn't want to go about
setting all this up because it's a lot of extra
work so i'm just putting it straight in in the
instructions which is again fine for the demo
but not recommended if you're doing a production
run on this so i put in my information my email
and my passport and then step four i'm basically
saying select the app engine studio go to this
hr thing under your app select hrdv everything
that i showed you earlier select preview And
then search for the name that's being asked of
you and click on that name in the first name
column. Increase the hours booked by the field
hours and decrease the hours left by the hours.
So we're basically updating the number of hours
in there. And then ultimately we are clicking
an update. We don't want it to ask for permission
because it has a tendency to ask for permission
and then it kind of like stops and cancels out.
And then once the update is complete, Tell me
how many hours have been booked so far and how
many are left. All right. So these are kind of
all my instructions. If you guys are interested,
I can put these in the comments or in the description
somewhere so you have a copy of them. But that's
basically what I'm telling it to do. Now, if
I scroll down, there's some other sections of
this that are important. We looked at the stored
credentials. You can turn on allowing specific
websites and desktop apps so that computer use
is only limited to go to certain places. If you
toggle this on, then you can say, look, from
a website, you can go to this website, but you
can't. So you can kind of like tell it which
ones it's allowed as well as applications. We're
leaving all that blank. And then from an inputs
perspective. So if you remember in our topics,
we actually have that person's name as a variable
called name. and hours is hours and so here i'm
going to go and create some inputs and i'm going
to say name name of the person click done and
then add another input and i'm going to put hours
number of vacation hours to book so these are
my inputs so you can see that they just got put
in here hours and name And I think at this point
in time, we're in good shape. So on the top right
hand side, I'm going to go ahead and click save.
Okay. Now, once this gets saved, you can actually
go ahead and click test over here and test it
out. But we're not going to do that. I'm just
going to run it inside the agent and show you
guys exactly what happens. Now, what we've done
so far is we created this topic. And we added
the tool of computer use with all these instructions,
right? Now what we're going to do is we're going
to select overview and we're going to go back
and we're going to give this agent instructions.
We're going to come to this instruction section
and we're going to select edit and we're going
to say, look, first use this. I'm going to put
slash and I want to use this topic, booking vacation
topic to find out for whom and how many hours
are being asked to book then use slash computer
use to service now the tool that we just added
the computer use to make the changes to the service
now database okay very simple save it and i think
with that That's our agent that we just built
using topics and the computer use. And you can
see knowledge is blank. The tool is just that
one tool. There is no triggers. However, autonomous
is actually recommended for computer use, probably
because computer use takes a long time and you'll
see in a second. And then in terms of agents,
like multi -agent orchestration, that's blank
as well. So there's a bunch of sections blank
in this, but that's okay. So what we're going
to do now is we're going to select test. And
then here I'm going to type in, I'd like to book
vacation and see what happens. So first this
computer use comes in, it comes in progress.
Then my booking vacation topic gets initiated.
And as you can see, it says, what is the first
and last name and so i can pick these names that
we put in let's put sammy belong and it says
how many hours you want to book and i say i want
to book 10 hours okay and now this is when the
computer use is going to start and it says that
computer use will begin as soon as a computer
is available it's a hosted computer so let's
give it a second and see what happens here Now
you can see that right away there are screenshots
that are being generated of what this computer
use is doing and it's telling you step by step
what it's doing. It's going to follow all the
instructions that we gave it. So first it's opening
a browser to access ServiceNow. It's going to
File Explorer and then opening the browser. So
it's going to kind of give you a play -by -play
and it looks like it's actually in the browser
now. It's going to go to that website. and this
does take a little bit of time that's why autonomous
is recommended for these types of computer use
tools but we'll see how it does as time goes
by all right looks like so far it's actually
made it to that service .developer site which
is good and now it's clicking in on the sign
in on the portal okay so it looks like it actually
got past this sign -in process over here and
so now it's actually following the rest of the
steps it's trying to get itself to this app engine
studio section so so far so good it's continuing
on and it's giving us a play -by -play as it
goes on Alright, looks like it's making good
progress. It actually made it all the way to
this database section where Jane and Pete and
everyone's there. And so now it's trying to get
to Sammy Belong's information here. And it's
updating the hours. it says updating hours left
to 80. all right so this is it this is the final
section and it says sami's hours already booked
has been updated to 120 and hours left to 80
and then the computer task is finished okay so
we're looking for 120 and 80 for sami so if we
go to our database and this is a moment of truth
right so it says right now it's 110 and 90 so
this should go up to 120 and this should go down
to 80. let's let me refresh the screen see what
happens here boom that's awesome it worked Maybe
I shouldn't be surprised. Well, that's awesome.
I'm happy about that. And if we kind of close
this out and go back, this is one of the issues
that I see with computer use, right? Now, people
are not that patient, you know, maybe a couple
of seconds. And you can see that this booking
vacation, the topic took five seconds, but this
took 283 seconds, essentially, right? It's a
long time. And that's why it's recommended that
autonomous is being used for computer use. So
think about like autonomous things that you can
do versus a real time thing that could take a
long time like that. So that's essentially it.
But one thing I wanted to mention as well is
I actually had to go to the settings and I went
down to this section here, which I showed before
this content moderation level. And I brought
it down to low instead of high. And then I ran
it. And I don't know if I showed that earlier
on, but experiment with that from high to low
if you can get your computer used to work. And
it might be because I actually had the username
and password in my instructions. If you enjoyed
this video, make sure to give it a like and consider
subscribing. Drop any questions you have in the
comments and I'll catch you on the next one.