Working with files in your Microsoft Teams bot (Preview)

We announced at Build 2018 that bots will soon be able to send and receive files. I’m happy to let you know that it’s finally here! If you have a Microsoft Teams app that works with files, it’s time to switch to Public Developer Preview and check it out.

 

To help get you started, I’ll explain how the feature works, and walk through a sample app that shows it in action. The bot uses the Azure Computer Vision API to recognize text in an image, then sends the text back to the user as a file. If you want to dive straight into code, the source for the sample is up on Github for both Node.js and C#.

 

ocrbot.pngSample conversation with OCR Bot

Receiving files

 

Your bot can now receive file attachments in a 1:1 chat (previously, they were silently dropped from the message activity). The attachment JSON looks like this:

 

{
    "contentType": "application/vnd.microsoft.teams.file.download.info",
    "contentUrl": "https://<onedrive_path>/phototest.tif",
    "name": "phototest.tif",
    "content": {
        "downloadUrl": "https://<onedrive_download_url>",
        "uniqueId": "70D29F4A-05B7-434A-B7E6-B651FBFEF508",
        "fileType": "tif"
    }
}
  • contentType is application/vnd.microsoft.teams.file.download.info.
  • name is the name of the file.
  • contentUrl is a direct link to the file on OneDrive for Business. Note that this is not how your bot will access the file.
  • content.fileType is the file type, as deduced from the file name extension.
  • content.downloadUrl is a pre-authenticated link to download the file.

To fetch the contents of the file, send a GET request to the URL in content.downloadUrl. The URL is only valid for a few minutes, so you must fetch the file immediately.

 

Inline images

Users have long been able to send an image to a bot by inserting it directly into the compose box. On desktop and web clients, the user copies the image content and pastes it into the compose box. On mobile, there is also a button to insert a picture from the photo library. Images sent this way are received as a different kind of attachment:

 

{
    "contentType": "image/*",
    "contentUrl": "https://smba.trafficmanager.net/amer-client-ss.msg/v3/attachments/<id>/views/original"
}
  • contentType starts with image/.
  • contentUrl is a resource under the Bot Framework /v3/attachments API.

 

OCR Bot

OCR Bot looks for an image in the incoming message, whether it's a file attachment or inline. Notice that we get the image content differently in the two cases. For a file attachment, the bot can pass the pre-authenticated downloadUrl directly to the Computer Vision API. However, because the contentUrl of an inline image requires an access token, the bot downloads the image bytes first, and then sends them as input to the API.

 

// 1) File attachment: a file picked from OneDrive or uploaded from the computer
const fileAttachments = msteams.FileDownloadInfo.filter(session.message.attachments);
if (fileAttachments && (fileAttachments.length > 0)) {
    // Image was sent as a file attachment
    // downloadUrl is an pre-authenticated URL to the file contents, valid for only a few minutes
    const resultFilename = fileAttachments[0].name + ".txt";
    this.returnRecognizedTextAsync(session, () => {
        return this.visionApi.runOcrAsync(fileAttachments[0].content.downloadUrl);
    }, resultFilename);
    return;
}

// 2) Inline image attachment: an image pasted into the compose box, or selected from the photo library on mobile
// getFirstInlineImageAttachmentUrl returns the contentUrl of the first attachment with a contentType that starts with "image/"
const inlineImageUrl = utils.getFirstInlineImageAttachmentUrl(session.message);
if (inlineImageUrl) {
    // Image was sent as inline content
    // contentUrl is a url to the file content; the bot's access token is required
    this.returnRecognizedTextAsync(session, async () => {
        const buffer = await utils.getInlineAttachmentContentAsync(inlineImageUrl, session);
        return await this.visionApi.runOcrAsync(buffer);
    });
    return;
}

 

Sending files

 

Similarly, your bot can send the user a file in 1:1 chat. There are several steps:

 

1) Request permission to upload the file

First, ask the user for permission to upload a file by sending a file consent card. For example:

{
    "contentType": "application/vnd.microsoft.teams.card.file.consent",
    "name": "result.txt",
    "content": {
        "description": "Text recognized from image",
        "sizeInBytes": 4348,
        "acceptContext": {
            "resultId": "1a1e318d-8496-471b-9612-720ee4b1b592"
        },
        "declineContext": {
            "resultId": "1a1e318d-8496-471b-9612-720ee4b1b592"
        }
    }
}
  • contentType is application/vnd.microsoft.teams.card.file.consent.
  • name is the proposed file name.
  • content.description is a description of the file.
  • content.sizeInBytes is the approximate file size in bytes.
  • content.acceptContext and content.declineContext are values that will be sent to the bot if the user accepts or declines, respectively.

To help the user decide, we recommend providing name, description and size.

 

OCR Bot

The code for sending the file consent card is in the returnRecognizedTextAsync function. When we detect text in the image, first we save the result in conversation data, assigning it a randomly-generated result id. We’ll need the text later, when the user accepts the file. We include the result id in the consent card’s context to associate it with a specific OCR result.

 

consentcard.pngOCR Bot sends a file consent card

// Save the OCR result in conversationData, while we wait for the user's consent to upload the file.
const resultId = uuidv4();
session.conversationData.ocrResult = {
    resultId: resultId,
    text: text,
};

// Calculate the file size in bytes. Note that this only needs to be approximate, 
// so if it's expensive to determine the file size, you don't need to do that.
// In this case it's straightforward to get the actual size, so we might as well. 
const buffer = new Buffer(text, "utf8");
const fileSizeInBytes = buffer.byteLength;

// Build the file upload consent card
// Accept and decline context contain the result id, to detect the case where the user acted on a stale card
const fileConsentCard = new msteams.FileConsentCard(session)
    .name(filename || "result.txt")
    .description("OCR result")
    .sizeInBytes(fileSizeInBytes)
    .context({
        resultId: resultId,
    });

// Send the text prompt and the file consent card.
// We send them in 2 separate activities, to be sure that we can safely delete the file consent card alone.
session.send("I found text in %s", result.language);
session.send(new builder.Message(session).addAttachment(fileConsentCard));

Note that the sizeInBytes value is just informational, and it can be approximate. In this case we already have the text, so it’s straightforward to calculate the exact size. But if it’s expensive to create the file, you can opt to include the necessary information in acceptContext, and defer generating the content until after the user accepts the upload.

 

2) User accepts or declines the file

When the user presses either “Accept” or “Decline”, your bot will receive an invoke activity.

 

{
    "type": "invoke",
    "name": "fileConsent/invoke",
    ...
    "value": {
        "type": "fileUpload",
        "action": "accept",
        "context": {
            "resultId": "82af7356-d2ef-4430-8a48-1ee189ca01db"
        },
        "uploadInfo": {
            "contentUrl": "https://<onedrive_url>/result.txt",
            "name": "result.txt",
            "uploadUrl": "https://<onedrive_upload_url>",
            "uniqueId": "03AA04A0-4D33-4760-94C4-FEE498B68FF2",
            "fileType": "txt"
        }
    },
    "replyToId": "1:xxxxxxx”
}
  • name is always fileConsent/invoke.
  • value.type is fileUpload.
  • value.action is accept if the user allowed the upload, or decline otherwise.
  • value.context is either the acceptContext or the declineContext in the card, depending on whether the user accepted or declined.

If the file was accepted, value contains an uploadInfo property with the following information:

  • name is the name of the file. Note that this may be different from the name that the bot proposed initially.
  • fileType is the file type, as determined by OneDrive.
  • contentUrl is a direct link to the final location of the file on OneDrive.
  • uploadUrl is the upload URL of the file. This points to a OneDrive upload session for the file. The upload session is valid for 15 minutes.

To set the file contents, issue PUT requests to the URL in uploadInfo.uploadUrl, as described in the documentation. If your bot cannot complete the upload—for example, if it hits an error while generating the file—it’s good practice to cancel the upload session by sending a DELETE request to the upload URL.

 

Your bot has 10 seconds to respond to the invoke message. If it might take longer than that to generate and upload the file, acknowledge the invoke by returning a successful HTTP response (e.g., 202 Accepted), and then finish the upload asynchronously. Keep in mind that the user is likely waiting for the file, so remember to send updates!

 

3) Send the user a link to the uploaded file

After you finish the upload, we recommend sending a link to the file. A file info card works well, as the user can click on the card to open the file directly in Teams.

 

{
    "contentType": "application/vnd.microsoft.teams.card.file.info",
    "contentUrl": "<uploadInfo.contentUrl>",
    "name": "<uploadInfo.name>",
    "content": {
        "uniqueId": "<uploadInfo.uniqueId>",
        "fileType": "<uploadInfo.fileType>",
    }
}
  • contentType is application/vnd.microsoft.teams.card.file.info.
  • The other fields are taken from the uploadInfo value received with the fileConsent/invoke message.

 

4) (Optional) Delete or update the file consent card

To prevent the user from acting on a file consent card multiple times, you can delete or update the consent card. The replyToId property of the fileConsent/invoke activity contains the activity id of the file consent card. Use this id in the corresponding APIs to delete or update the activity.

 

OCR Bot

The bot processes the file consent invoke activity in handleFileConsentResponseAsync. First, it determines from value.action whether the user accepted or declined. If the user declined, we simply acknowledge the action, and delete both the file consent card and the OCR result. If the file was accepted:

  1. First, we check that the user acted on a consent card that corresponds to the current result. This helps guard against the user acting again on a previous card. In this case this is more of a precaution, because we proactively delete the consent card anyway.
  2. Then, we upload bytes into the file. It’s important to follow the rules of the OneDrive upload session, otherwise you may receive errors. We have the full text ready, so we can upload the entire content in one go.
  3. If the upload succeeds, we send a file info card. The card is populated based on information in the uploadInfo object.
const lastOcrResult = session.conversationData.ocrResult;

// Delete OCR result and file consent card
delete session.conversationData.ocrResult;
const addressOfSourceMessage: builder.IChatConnectorAddress = {
    ...event.address,
    id: event.replyToId,
};
session.connector.delete(addressOfSourceMessage, (err) => {
    if (err) {
        console.error(`Failed to delete consent card: ${err.message}`, err);
    }
});

const value = (event as any).value as msteams.IFileConsentCardResponse;
switch (value.action) {
    // User declined upload
    case msteams.FileConsentCardAction.decline:
        session.send("File upload declined");
        break;

    // User accepted file
    case msteams.FileConsentCardAction.accept:
        const uploadInfo = value.uploadInfo;

        // Check that this response is for the the current OCR result
        if (!lastOcrResult || (lastOcrResult.resultId !== value.context.resultId)) {
            session.send("Result has expired ");
            return;
        }

        // Upload the file contents to the upload session we got from the invoke value
        const buffer = new Buffer(lastOcrResult.text, "utf8");
        const options: request.OptionsWithUrl = {
            url: uploadInfo.uploadUrl,
            body: buffer,
            headers: {
                "Content-Type": "application/octet-stream",
                "Content-Range": `bytes 0-${buffer.byteLength-1}/${buffer.byteLength}`,
            },
        };
        request.put(options, (err, res: http.IncomingMessage, body) => {
            if (err) {
                session.send("File upload error: %s, err.message);
            } else if ((res.statusCode === 200) || (res.statusCode === 201)) {
                // Send message with link to the file.
                const fileInfoCard = msteams.FileInfoCard.fromFileUploadInfo(uploadInfo);
                session.send(new builder.Message(session).addAttachment(fileInfoCard));
            } else {
                session.send("File upload error: %s", res.statusMessage);
            }
        });
        break;
}

 

Caution: developers at work 👷

 

For now, the ability to receive file attachments and send files to the user is restricted to 1:1 chats with the bot. File attachments will continue to be dropped silently for messages sent to bots in channels and group chats. We know that working with files is a useful capability, and we are working on extending this mechanism to those contexts as well.

 

You might also run into some known issues. (We’ll get these fixed before we make the feature generally available.)

  • The file consent card shows only the file name, omitting the description and file size.
  • If the bot sends a different file with the same name as one that already exists, the previous file is replaced. The user has no opportunity to select a new name.
  • If the bot abandons an upload session for an existing file, then subsequent attempts to click on “Accept” could fail silently.

 

Finally, note that this is still in Public Developer Preview, and the exact API and UX could change. That depends in part on feedback that we receive from our developer community—yes, that’s you! So please check out our Node.js and C# samples, try using the feature in your own app, then let us know what you think. You can reach us through our various developer support channels: Github, Stack Overflow, and email. We look forward to hearing from you.

 

Happy filing!

By |2018-06-04T19:50:19+00:00June 4th, 2018|Uncategorized|Comments Off on Working with files in your Microsoft Teams bot (Preview)