Using existing models

You can interact with existing models in all sorts of ways! You might…

In this section we’ll explain what AI and machine learning models are capable of, show you a quick interactive demo of each, and give you snippets of code to make them come alive on your own.

What type of models are you most interested in?

Text-based models

Text-based models can classify and extract information out of documents.

 
Text models overview
While working with image classification is probably the most common task you’ll find, there are plenty of other more advanced possibilities as well!

Text classification
Text classification is one of the most basic yet most useful machine learning methods: it’s the process of putting pieces of text into categories.
 
Detecting people, places, things and more
Named entity recognition (NER) is also known as token classification or span classification. Instead of classifying an entire document, it pulls out bits and pieces you might be interested in: names, organization names, places, money, etc.

Summarization
When you want to summarize something, clearly!

Asking questions about documents
Question answering has a lot of sub-sections, but the idea in general is that you have a document and ask the model a question to be answered from it.
 
Document similarity
Document similarity or sentence similarity can be used to find pieces of text that are related to one another. Through a bit of magic, each document is transformed into 384 dimensions, which can be used to find ones that are about the same topics. The dimensions don’t mean anything, but you can read more about word embeddings here.

Asking PDFs questions
Asking PDFs questions actually isn’t about text, it’s about images! Except… it’s kind of about text. And layout. And images. It’s complicated.
No matching items

Image-based models

Image-based models can analyze objects that appear in photographs and satellite imagery.

Image models overview
There are many different tasks you can pursue with images! The most interesting concept you’ll find is “stuff” vs “things,” which you can explore when reading about semantic segmentation and instance segmentation.

Image classification
Image classification allows you to put images into category. Typically these are yes/no questions.

Object Detection: Finding things in photos
There are a handful of different ways that computers can look at images, and one of the options is “stuff” versus “things.” Instance segmentation can be used to find “things:” each individual person, car, tree, etc in an image.

Semantic Segmentation: Finding stuff in photos
There are a handful of different ways that computers can look at images, and one of the options is “stuff” versus “things.” Semantic segmentation can be used to find “stuff,” the number of pixels dedicated to people, cars, vegetation, etc in an image.

VQA: Asking questions about pictures
Visual Question Answering (VQA) is a way to ask questions about pictures! If we want to get technical, it isn’t actually an image model: it’s a multimodal model that understands both images and text! The text part comes in when understanding what your question means.

Image description
There are a handful of different ways that computers can look at images, and one of the options is “stuff” versus “things.” Instance segmentation can be used to find “things:” each individual person, car, tree, etc in an image.
No matching items

Audio models

Audio models can translate and transcribe audio from a multitude of languages.

Video models

Video models can wrangle video in unimaginable ways (yes, this is an awful description, but they get pretty wild prettyy fast).