Hey everyone and welcome back to the second and final part of this tutorial series where we explore the intricacies of Apple’s MusicKit by building our very own music player in SwiftUI that can stream songs from our Apple Music account. If you haven’t read Part 1, you can do that here.

In the last tutorial, we looked at how to create a MusicKit Identifier for our Apple Developer account, created a JSON Web Token private key, and we were able to successfully make web requests to the Apple Music API. At the end of the tutorial, we also built the UI of our music player and were able to make it look like below.

In this tutorial, we’ll start making calls to the Apple Music API to populate our app with real data. We’ll also look at how to control media playback with the MediaPlayer framework and enhance the app to this.

Cool, right? Let’s get started.

Note: This tutorial was made using Xcode 11.4 and Swift 5.1. An active Apple Music subscription will be needed in order to test the app out. You will also need to run your app on a real device because as of this time, the Simulator does not support Apple Music playback.

Apple Music API vs iTunes Search API

If you’ve worked with podcasts, tv shows, or any other Apple generated media content, you must have come across the iTunes Search API. The iTunes Search API lets you search for content in the iTunes Store, App Store, and iBooks Store. This will give you access to information about apps, books, movies, podcasts, music, music videos, audiobooks, and TV shows. The best part about using this API is that it’s completely free so you can get updated information, on the fly, for no additional cost.

But why do we use the Apple Music API instead of the iTunes Search API? This is because of how much information and power the Apple Music API can deliver. While the Apple Music API provides the same functionality as the iTunes Search API (fetching information about music and music videos), the Apple Music API takes it to the next level by accessing a user’s personal library and recommendations.

This means your app can leverage a user’s playlists, songs, or ratings for their content. This is why it’s required for your user to have an Apple Music account if you will be using the API. Another important reason why the Apple Music API is better over the iTunes Search API is because it is built to work harmoniously with the MediaPlayer framework. As you’ll see, streaming audio is a breeze with the Apple Music API. Let’s get started!

Creating our Classes

Due to the nature of SwiftUI, making URL calls within our structs can get really complicated. Furthermore, it can get pretty complicated to populate our UI with different information gathered from different songs. This is why we’ll create an AppleMusicAPI class that will contain the functions we’ll frequently use in our app and a Song structure to make it easier to populate our UI.

Before we begin, download SwiftyJSON.swift and add the file to your project. This file contains the code for the popular library SwiftyJSON. This is a tremendous tool when making HTTP requests and receiving and parsing JSON responses. You can find more information about SwiftyJSON here.

To begin, choose to File > New > File. From the popup that appears, select Swift file. Name this file AppleMusicAPI.swift and save this file in the default location. You should now have a template file as shown below.

Similarly, let’s create a struct for our songs. Go to File > New > File and select Swift file. Name this file as Song.swift and save this file. We’ll begin creating our Song structure first.

Add the following code below import Foundation:

struct Song {
    var id: String
    var name: String
    var artistName: String
    var artworkURL: String

    init(id: String, name: String, artistName: String, artworkURL: String) {
        self.id = id
        self.name = name
        self.artworkURL = artworkURL
        self.artistName = artistName
    }
}

The code above creates a simple structure for Song. In case if you’re not familiar with structures, don’t worry. A structure is a type of class that helps make your code more concise, especially when dealing with structural data. Structures, just like classes, can define properties, methods, and initializers to set up their initial state. This is what we are doing with our init method. We are providing a default initializer method for the Song structure that lets a user input the ID, name, artist’s name, and album artwork’s URL.

Now switch over to the AppleMusicAPI.swift file. Here, we’ll be implementing a class called AppleMusicAPI that stores our developer token and a bunch of methods that will help when playing our song through our app.

Remove import Foundation and Add the following code to AppleMusicAPI.swift:

// 1
import StoreKit

// 2
class AppleMusicAPI {
    // 3
    let developerToken = "YOUR DEVELOPER TOKEN FROM PART 1"

    // 4
    func getUserToken() -> String {
        var userToken = String()

        return userToken
    }
}

The above code is pretty self-explanatory but don’t worry if this is a little unclear. Here’s what the above function does.

1. First, we import the StoreKit framework. This helps us access a lot of built-in methods that can communicate with the Apple Music API.
2. Here, you’ll notice we’re defining a class called AppleMusicAPI. This makes it easier to manage multiple instances of this class and reference the methods from other views in our app.
3. Here we’re defining a constant developerToken containing the developer token we created in Part 1 on this tutorial series. This makes it easier to call the token when communicating with the API.
4. Finally, we define our first method in this class: getUserToken(). As mentioned earlier, the Apple Music API can get a user’s library and playlists. This is only possible if we receive a token that is identifiable to a particular user.

Let’s finish implementing the rest of the getUserToken method.

func getUserToken() -> String {
    var userToken = String()

    // 1
    let lock = DispatchSemaphore(value: 0)

    // 2
    SKCloudServiceController().requestUserToken(forDeveloperToken: developerToken) { (receivedToken, error) in
        // 3
        guard error == nil else { return }
        if let token = receivedToken {
            userToken = token
            lock.signal()
        }
    }

    // 4
    lock.wait()
    return userToken
}

You might face some new code in the above snippet. Don’t worry though as most of it will be explained.

1. First, we define a lock that is of type DispatchSemaphore. What is a DispatchSemaphore? A dispatch semaphore is an efficient implementation of halting a thread until a particular message has been passed. This “locks” a thread from executing more code until a signal has been given.
2. We access the SKCloudServiceController().requestUserToken() method to get a token that authenticates the user in personalized Apple Music API requests. Notice how we use our constant developerToken in this method. It’s definitely easier than typing the long string again and again.
3. Here, we write some code to error check what the requestUserToken() function returns. If receivedToken is not empty, we set it equal to our userToken variable from above. Notice, afterwards, we call lock.signal(). This lets the dispatch semaphore we create earlier know that it’s ok to start executing remaining code.
4. Since this method executes asynchronously, it’s possible that when getUserToken() executes, it can skip to the line return userToken, even before a token is received from the SKCloudServiceController. By adding the lock.wait() line of code, this tells the code to halt executing any further code until a signal is given (as we implemented in Step 3).

Note: Dispatch sophomores must be used with caution. This can transform code to execute synchronously which can slow down an app or not update UI in time. In fact, if lock.signal is not called, the app will forever remain stuck until the user restarts the app. Thus, it’s not advised to use this in big production apps. However, for our purposes, it’s completely acceptable.

Now, let’s implement our next method: fetchStorefront().

A storefront is an object that represents the iTunes Store territory that the content is available in. When we perform a search using the Apple Music API, we’d like to show results relevant to our user’s location. Beneath, getUserToken(), add the following method:

func fetchStorefrontID() -> String {
    // 1
    let lock = DispatchSemaphore(value: 0)
    var storefrontID: String!

    // 2
    let musicURL = URL(string: "https://api.music.apple.com/v1/me/storefront")!
    var musicRequest = URLRequest(url: musicURL)
    musicRequest.httpMethod = "GET"
    musicRequest.addValue("Bearer \(developerToken)", forHTTPHeaderField: "Authorization")
    musicRequest.addValue(getUserToken(), forHTTPHeaderField: "Music-User-Token")

    // 3
    URLSession.shared.dataTask(with: musicRequest) { (data, response, error) in
        guard error == nil else { return }

        // 4
        if let json = try? JSON(data: data!) {
            print(json.rawString())
        }
    }.resume()

    // 5
    lock.wait()
    return storefrontID
}

1. Just like earlier, we create a dispatch semaphore called lock to make sure that the function returns a storefrontID only after the data has been received from our URL request.
2. We create a URLRequest using the Apple Music API url. A lot of the details about handling requests and responses from the Apple Music API can be found here, but here’s the gist. To compose a request to the API, first specify the root path, https://api.music.apple.com/v1/  . Here, we specify the storefront component. Using a GET Request, we create a request, adding our developer token in the header.
3. Next, we use URLSession to send musicRequest. This is very similar to performing any networking requests for either obtaining images or data. As the .dataTask method is built, we are returned with three constants: data (the data the network response sends back), response (details about the response), and error(this may be nil if there is no error). After checking to make sure there is no error, we move to Step 4.
4. The data we get is a JSON payload. A payload is the part of transmitted data that is the actual intended message. Using the SwiftyJSON library we added earlier, we’ll print the raw JSON output before parsing it. Notice that we’re not calling lock.signal() here because we’re not setting anything to the storefrontID. If we did call lock.signal(), our app would crash when testing it. We’ll add the signal when we parse our JSON result.
5. Finally, we ask the dispatch semaphore to wait before returning the storefront’s ID.

This was a lot of code so now is a good time to make sure everything is working properly. We’ll switch to ContentView.swift and add an onAppear function to our TabView. Whatever code we place inside this method will be executed when TabView is displayed to the user. The code we’ll place inside will ask the user for permission to access their media library and upon authorization, will call the fetchStorefront() method which will print the JSON payload.

Let’s do it! At the top of the file, add import StoreKit and then make your TabView view look as such

TabView(selection: $selection) {
// Previous Code
    ...
}
.accentColor(.pink)
.onAppear() {
    SKCloudServiceController.requestAuthorization { (status) in
        if status == .authorized {
            print(AppleMusicAPI().fetchStorefrontID())
        }
    }
}

These few lines of code are easily comprehendible. We ask the SKCloudServiceController (remember, this is an object provided by StoreKit that determines the current capabilities of the user’s music library) for authorization to access a user’s media library. If the user authorizes this access, we run print(AppleMusicAPI().fetchStorefrontID()) which will print the JSON payload of the request.

Notice how we call out AppleMusicAPI class. Similar to ContentView in SwiftUI or ViewController in storyboard-based Swift, by adding the pair of parentheses after the class’s name, we are initializing it which gives us access to all its constants and methods, such as the fetchStorefrontID() method.

Now before we run the app to make sure everything is functioning as expected, we need to make a slight addition to our Info.plist. We need to add the following property that gives gives a description to the user what we need access to their music library for. The key is Privacy - Media Library Usage Description and the value can be any message you wish it to be that adequately informs users what the data will be used for.

Now we can run out app!

You’ll need to run the app on a physical device with an active Apple Music subscription. Unfortunately, the simulator does not have the Music app so accessing a user’s Apple Music account is near impossible. Having an active Apple Music subscription lets you test the features of your app on your account.

After authorizing the app to access your Apple Music account, you should wait for some time and expect an output that looks like the one below.

Don’t worry about the error messages stating something along the liens of “Unable to get the local account”. Beneath that, you’ll see a JSON string printed out. This goes to show that our code is working. Let’s go back to AppleMusicAPI.swift and finish the rest of the fetchStorefrontID() method. Delete the line print(json.rawString()) and replace it with the following

// 1
let result = (json["data"]).array!
let id = (result[0].dictionaryValue)["id"]!

// 2
storefrontID = id.stringValue

// 3
lock.signal()

1. We set result to be the array the JSON payload provides underneath the data key. id is simply value provided under the id key in the dictionary provided in result. This can sound a little confusing. The names of keys and values can be derived from the JSON string we printed earlier. To learn more about reading and parsing JSON, this [article][4] can help you.
2. We set the storefrontID to the string value of the id constant from Step 1.
3. Just like earlier, we signal the dispatch semaphore that it’s okay to execute remaining code and free up the thread.

Your entire fetchStorefrontID() method should look like this.

Run the app again and you’ll see how instead of printing a JSON file, only two characters are printed, signifying the storefront ID of your Apple Music account. For me, in the United States, a simple “us” is printed out.

If everything works, give yourself a pat on the back! We have one last method to implement in our AppleMusicAPI class and this one is pretty big. This method will allow us to search Apple Music’s entire library of 40 million+ songs based on the keywords given. Now’s a good time to take a break!

Music Search

Underneath our fetchStorefrontID() method, let’s add a new method called searchAppleMusic(searchTerm:) that will let us search Apple Music’s library based on the term. Here’s a boilerplate to get you started. You’ll notice it’s very similar to our fetchStorefrontID() function.

func searchAppleMusic(_ searchTerm: String!) -> [Song] {
    let lock = DispatchSemaphore(value: 0)
    var songs = [Song]()

    let musicURL = URL(string: "https://api.music.apple.com/v1/catalog/\(fetchStorefrontID())/search?term=\(searchTerm.replacingOccurrences(of: " ", with: "+"))&types=songs&limit=25")!
    var musicRequest = URLRequest(url: musicURL)
    musicRequest.httpMethod = "GET"
    musicRequest.addValue("Bearer \(developerToken)", forHTTPHeaderField: "Authorization")
        musicRequest.addValue(getUserToken(), forHTTPHeaderField: "Music-User-Token")

    URLSession.shared.dataTask(with: musicRequest) { (data, response, error) in
        guard error == nil else { return }

    }.resume()

    lock.wait()
    return songs
}

In terms of variables, we have our routine dispatch semaphore defined and a new songs array of type Song that we defined earlier. We’ll be storing the songs in this array and returning this array to populate our table. You’ll notice that searchTerm is a String provided as an input to this function. We also perform string manipulation on searchTerm to replace any instances of whitespaces with the + symbol as URL’s can’t contain any whitespaces.

We also create the URLRequest using the Apple Music API url. Notice that like before, it uses the root path https://api.music.apple.com/v1/. However, the URL component is catalog and we pass the search term as a parameter of the URL. Apart from these changes, we again use the GET request, create the request, and add our developer token to the header.

Now comes the important part which is parsing the data returned from our URLSession.shared.dataTask. Let’s think about what we need. We need to parse the data for an array of songs. We then need to isolate the different components of the each song in the array such as title, artist, album artwork, etc. After isolating these components, we can create an object of type Song and add this to our songs array. This is how it can be accomplished.

Type the following code and place it underneath guard error == nil else { return } inside the URLSession.shared.dataTask() method:

// 1
if let json = try? JSON(data: data!) {
    // 2
    let result = (json["results"]["songs"]["data"]).array!
    // 3
    for song in result {
        // 4
        let attributes = song["attributes"]
        let currentSong = Song(id: attributes["playParams"]["id"].string!, name: attributes["name"].string!, artistName: attributes["artistName"].string!, artworkURL: attributes["artwork"]["url"].string!)
        songs.append(currentSong)
    }
    // 5
    lock.signal()
} else {
    // 6
    lock.signal()
}

This is very similar to the code we wrote above to fetch a user’s storefront ID. The only difference it there’s a lot more JSON parsing and we are creating a Song object to add to our array.

1. First, we type check to see is the data returned is in a valid JSON format. If it is, we create a constant called json containing all this data in the JSON file format.
2. We parse json for the songs that is nested within results -> songs -> data in an array format. Now, I don’t want to burden you with more JSON formatting but this order was derived by printing the raw string of json and noticing how each component was ordered. If you want, you can print json to see how result is nested within the JSON data. result will now contain an array of our songs.
3. Now, we parse through each song in result using our trusty for loop.
4. First, we define a constant called attributes that is a dictionary of all the attributes of our current song. These attributes contain lots of information about the current song but we’re interested in the id, name, artist’s name, and artwork URL. Then we create a currentSong constant of type Song that is populated with the values from the attributes dictionary. Finally, we append the currentSong to our songs array.
5. Last but not least, we signal our dispatch semaphore throughlock so the rest of the thread can be freed up to run the remaining code.
6. This is more of a check but if the data returned is not in a valid JSON format, we don’t want our app to stay stuck forever. This is why we add an else clause to signal the dispatch semaphore to continue the rest of the code.

Now we need to make sure our function works. Head over to ContentView.swift. In the .onAppear() declaration, we need to make a very minor change to the statement where we print AppleMusicAPI().fetchStorefrontID(). All we need to do is replace .fetchStorefrontID() with .searchAppleMusic("Taylor Swift"). If all goes well, when we run the app, this will print an array of Song objects containing all the songs Apple Music returns when its catalog is searched for Taylor Swift. (Of course, you can replace this with any artist, album, song, or word of your choice).

Run the app and observe the console. After a few seconds, you should see an output similar to the following:

As you can see, a whole array containing the songs based on your search term. We can look at the artist name, id, name of the song, and a URL to the album artwork. Since we know our code works, let’s delete the entirety of the .onAppear() declaration. That way we don’t necessarily make calls to the API when our app loads.

Now comes the second part of this section which is populating our search table view with the results.

Populating the Table

Go to SearchView.swift. We’ll be making all the changes here to populate our search table. There are many changes we have to make so follow carefully.

First, we need to replace the songs array containing some temporary strings. Delete that line and replace it with the following line:

@State private var searchResults = [Song]()

This creates a new empty array called searchResults that conforms to our Song object. We also link it to the @State property so anytime any changes are made to this array, any UI using this variable will be updated.

Now, as expected, you’ll get some errors since we removed our old songs array. Let’s start at the ForEach within our List SwiftUI component. The error displayed here is that we’re still trying to iterate over a variable songs that doesn’t exist. Replace the ForEach line with the following:

ForEach(searchResults, id:\.id) { song in
    ...

There are three changes we’ve made here.

1. The first change is we’ve replaced songs which searchResults.
2. Now since our Song object doesn’t conform to the Hashable property, we can’t use .self to use it as an id. Good news though! Our Song objects have an id property to identify each element. This is why we replace \.self with \.id.
3. Finally, and this is really minor, we replace the songTitle variable to song. songTitle referred to the string from our old array. This isn’t very representative of the current array for which we are iterating which is why we rename it to song.

While this error is removed, we now have errors wherever we used the variable songTitle. This is a simple fix as we replace songTitle with song.name. Modify the code after the Image object within your HStack to look like the following:

VStack(alignment: .leading) {
    // 1
    Text(song.name)
        .font(.headline)
    // 2
    Text(song.artistName)
        .font(.caption)
        .foregroundColor(.secondary)
}
Spacer()
Button(action: {
    // 3
    print("Playing \(song.name)")
}) {
    Image(systemName: "play.fill")
        .foregroundColor(.pink)
}

1. first change we made was to replace songTitle with song.name. This will use the name of the song as the string value for this Text object.
2. The second change we made is to replace the ”Artist Name” placeholder with the actual artists name. Just like song.name, we use song.artistName to populate this Text object with the value in song.artistName.
3. Finally, the last change we made was a slight modification to our Button object which is to replace songTitle with song.name just like Step 1.

Now, you can build and run the app. Everything should compile and work as normal but there’s a catch. You’ll notice that when you search for a song and press enter, nothing happens. This is because we still haven’t called the searchAppleMusic function from our AppleMusicAPI() class. To do this, we need to make some changes to our TextField. Before this, at the top of the file underneath import SwiftUI, add the line import StoreKit.

Now, make the following changes to your TextField:

TextField("Search Songs", text: $searchText, onCommit: {
    // 1
    UIApplication.shared.resignFirstResponder()
    if self.searchText.isEmpty {
        // 2
        self.searchResults = []
    } else {
        // 3
        SKCloudServiceController.requestAuthorization { (status) in
            if status == .authorized {
                // 4
                self.searchResults = AppleMusicAPI().searchAppleMusic(self.searchText)
            }
        }
    }
})
.textFieldStyle(RoundedBorderTextFieldStyle())
.padding(.horizontal, 16)
.accentColor(.pink)

We’re at the last stretch here but this code shouldn’t be too complicated for you to read. Here’s a gist of what it compiles:

1. First, we’d like to dismiss the keyboard when the user presses Search on the keyboard. This is done by calling the line UIApplication.shared.resignFirstResponder() .
2. Next, we’d like to make sure that if the user doesn’t enter anything, our searchResults are empty. This is why we set it to an empty array if the searchText is empty.
3. If the searchText variable is not empty, we’d like to call our searchAppleMusic function. Before we do this, we make sure that SKCloudServiceController has the necessary authorization to access a user’s media library. If the user authorizes this access, we can continue to the next line of code.
4. Finally, we set out searchResults array equal to the results of AppleMusicAPI().searchAppleMusic(self.searchText). This will pass the search term to our searchAppleMusic function and update our searchResults array with whatever Apple Music returns!

We’re finally ready to run our app. Build and run the app and head over to the Search page. Enter any term, wait for a couple seconds as your app makes network requests you’ve implemented, and watch with delight as the table view is populated with the results of your search term.

Congratulations! You’ve mastered the basics of the Apple Music API! As you can see, it’s very repetitive in that the steps can be summed down to the following:

1. Make a network call to the Apple Music API
2. Parse the JSON to see where the data you need can be found
3. Create an object and populate it with the data you’ve parsed

For the remainder of the tutorial, we’ll be focusing on the under appreciated MediaPlayer framework and its very important object, MPMusicPlayerController . Also, don’t worry about the album artwork not displaying as of now. I’ll show you how we can use Swift Package Manager to successfully load and display images from a URL.

Implementing Music Play Back

Get excited because now we’ll be looking at the MediaPlayer framework and how to access and control your device’s music player. Essentially, the MediaPlayer framework is a part of MusicKit and lets you control playback of the user’s media from your app.

To play content using this framework, we have to instantiate one of the built-in MPMusicPlayerController objects. There are two types. Here is a brief description of what they are:

1. System Player: This media player is directly linked to the Music app on your device. If you choose to use this player, then when a user click on a song, it will open up Music and start playing from that app.
2. Application Player: This will be a media player that is built-in directly to your app. That way when a user clicks on a song, they won’t be transported to the Music app, but rather all the playback functionality will occur in your app. Of course, this means that you will have to add play/pause and skip/rewind functionality. We’ll be using this type of player for our app.

Let’s add the application player to our app. Now we want to modify this object within both screens of our app. In our Player View, we want to control the playback of the currently playing item in the song and in our Search View, we want to be able to play songs directly from that page. As such, we’ll be implementing @State and @Binding protocols to help with data flow.

Head over to ContentView.swift. At the top of the file, type import MediaPlayer underneath import StoreKit. Next, add the following line where you declare your selection variable:

@State private var musicPlayer = MPMusicPlayerController.applicationMusicPlayer

We’re creating a variable called musicPlayer that is of the type applicationMusicPlayer as we discussed above. We’re also adding the @State property to it so we can use it to pass it to other views. Let’s do that right now!

Go to PlayerView.swift and at the top of the file, add import the MediaPlayer framework with:

import MediaPlayer

Next, at the top of thestruct, add the following line:

@Binding var musicPlayer: MPMusicPlayerController

This creates a Binding variable for our Player View. Since we will be binding this variable to the musicPlayer we created in ContentView.swift, anytime we make a change in Player View to this object, it will be reflected in Content View.

Since we can’t pass this binding variable to our PlayerView_Previews and SwiftUI previews have no support for MusicKit, there’s no point in having this in our file anymore. Delete the entire PlayerView_Previews structure from this file.

Let’s do the same thing in SearchView.swift. You should already have the MediaPlayer framework implemented. Just like before, add the following line to the top of your SearchView structure below the initialization of the searchResults array.

@Binding var musicPlayer: MPMusicPlayerController

Similar to PlayerView.swift, delete the entire SearchView_Previews structure from this file.

Now, head back to ContentView.swift. You should see some errors for PlayerView() and SearchView(). This is simply because we haven’t added the argument for the parameter musicPlayer when we call it. You can fix this by changing PlayerView() to PlayerView(musicPlayer: self.$musicPlayer) and SearchView() to SearchView(musicPlayer: self.$musicPlayer).

Build the app! It should compile without errors. We now have a built-in media player that is accessible to all the views in our app! Now when we implement the play and pause methods, we can easily update our player and reflect those changes across all the views in our app!

Playing Music

When our user clicks on a song from the search table, we’d like our app to play the song selected. This means that we have to add a song to our musicPlayer’s queue and tell the queue to play. Watch how we implement this in two line!

Go to SearchView.swift. Find the Button structure that should be inside the List structure. As of right now, it only prints the name of the song it is playing to the console. Delete that line and replace it with the following:

self.musicPlayer.setQueue(with: [song.id])
self.musicPlayer.play()

That’s it! In the first line, we set the queue of our musicPlayer. The queue in this case is an array of strings containing the ID’s of the song. This is why the MediaPlayer and Apple Music API work harmoniously together. The framework can automatically detect which song is to be played based on the ID given. The final line simply instructs our musicPlayer to play whatever is in the queue!

Build and run the app! You should be able to search for a song, and when you tap on it, your music player should start playing the song!

If you’ve got everything working, congratulations! This is one of the most under appreciated, unknown functionalities you can build using Swift! We’re not done yet though! While we can play music, we’d also like to control the playback state of the song! We also want to make sure that the Player View can show us the name of the song and tag artist’s name. This can be achieved relatively simply as well!

Go to PlayerView.swift. Let’s start at the top of the file and make the respective changes as we work our way down!

Find the VStack structure where we define our Text objects that will contain the name of the song and the artist’s name. Replace it with the following:

VStack(spacing: 8) {
    Text(self.musicPlayer.nowPlayingItem?.title ?? "Not Playing")
        .font(Font.system(.title).bold())
    Text(self.musicPlayer.nowPlayingItem?.artist ?? "")
        .font(.system(.headline))
}

The changes we’ve made here is to replace “Song Title” and “Artist Name“ with the actual name of the currently playing song and the artist by whom it’s sung. If no song is playing, then we leave the artist’s name empty and set the name of the song to ”Not Playing”.

Before we go further, I’d like to explain a little bit about this .nowPlayingItem object we’re calling. This object is of a type called MPMediaItem. This object is provided by the MediaPlayer framework and is an object which contains a collection of properties that represents what you could call a “song” in a media library. More specifically, an MPMediaItem can be any object that can be played and has a metadata that is similar to any audio-based object such as a song or podcast. The properties we’re using here are title, which is the name of the song, and artist, which is the name of the artist. There are many more properties that you can use if you choose to that can be found here.

Build and run your app. You can see that when you start to play a song, the metadata is displayed where it’s supposed to be!

Now, let’s add some play and pause functionality. Scroll down in PlayerView.swift and go to the Button object which currently prints ”Pause” to the console. Delete this line and replace it with the following:

if self.musicPlayer.playbackState == .paused || self.musicPlayer.playbackState == .stopped {
    self.musicPlayer.play()
} else {
    self.musicPlayer.pause()
}

Our musicPlayer has a property called playbackState that is of type MPMusicPlaybackState. This property keeps track of whether the music player is playing a song, paused, or stopped. If the playbackState is equal to.paused or .stopped, then nothing is playing and when the user clicks on this button, we’d like for it to start playing. This is why we call self.musicPlayer.play(). Similarly, in the else clause, we call self.musicPlayer.pause() because we’d like to stop the currently playing song.

You can build and run the app on your device. You should see that when you click on the pause button, the music pauses and when you press again, the music resumes. However, the icon of the button is not updating. This can be done by tracking the playback state of the currently playing item!

At the top of PlayerView.swift, underneath where you declared musicPlayer, type the following line:

@State private var isPlaying = false

This is a Boolean variable that is attached to the @State property that will be updated based on the playback state of the currently playing song. First, go to the Button object we modified above. Modify the action of the Button to look like the following:

if self.musicPlayer.playbackState == .paused || self.musicPlayer.playbackState == .stopped {
    self.musicPlayer.play()
    // 1
    self.isPlaying = true
} else {
    self.musicPlayer.pause()
    // 2
    self.isPlaying = false
}

We’ve added two lines that are pretty self explanatory. When we signal the musicPlayer to play, we set the isPlaying variable to true. When we pause the musicPlayer, we once again set the isPlaying variable to false.

Next, we need to use the isPlaying variable to update the image used as the logo for the button. This is quite simple. Underneath the button’s action where we declare the UI of the button, modify the ZStack object to look as such:

ZStack {
    Circle()
        .frame(width: 80, height: 80)
        .accentColor(.pink)
        .shadow(radius: 10)
    Image(systemName: self.isPlaying ? "pause.fill" : "play.fill")
        .foregroundColor(.white)
        .font(.system(.title))
}

The change we made here is to change the systemNameof the Image object used based on whether self.isPlaying is true or false. If it is true, we’ll use the pause.fill SF Symbol. If not, we’ll use the play.fill symbol.

Last but not least, we need a way to update self.isPlaying if there is a change made in Search View. For example, if our player is paused but we play a song from Search View, we’d like to update self.isPlaying by setting it to true. Now, we can use the @Binding protocol but I’d like to show you an easier alternative: the onAppear() function!

At the very bottom of PlayerView.swift, find the second to last } closing curly bracket. This is the bracket that will close our GeometryReader. Attach the .onAppear() function below it as shown:

.onAppear() {
    if self.musicPlayer.playbackState == .playing {
        self.isPlaying = true
    } else {
        self.isPlaying = false
    }
}

What are we doing in this function? Well, remember that our musicPlayer is already bound to all the views in our app. So any updates to this object is immediately reflected in every view. When we play a song from Search View, the musicPlayer is updated to set its playbackState to .playing. This is why in the .onAppear() function, we check to see that if the playbackState is in fact playing, then we set isPlaying to true. If not, we set isPlaying to false!

That’s all for playing and pausing! Build and run your app! You should see that the play/pause button accurately changes based on whether the song is currently playing or is paused!

Implementing Skip & Rewind

Now let’s implement the logic for skip and rewind buttons. This won’t take too long. Let’s start with the skip button.

Navigate to PlayerView.swift and locate the Button object that we’re using as a temporary skip button. It should be printing ”Skip” to the console when tapped upon.

Replace the line print("Skip") with self.musicPlayer.skipToNextItem() and that’s all! Build and run your app. When you press the skip button, it will jump to the next song in the queue, which is nothing, so the player will stop!

You can see how MediaPlayer comes with a list of functions that make it easy for us to control playback! With one line of code, we were able to implement a “skip song” functionality into our music player app. The “rewind” button is a little more challenging.

Think about your favorite music player. When you press the rewind button, what happens? If the song is more than 5 seconds through its playback, it skips to the beginning of the song. If the current playback is still within the first 5 seconds, then it goes to the previous song. With a simple if-else statement, watch how we can add the same functionality to our app.

Scroll back up to the Button object that currently prints ”Rewind” to the console. Delete that line and replace it with the following:

if self.musicPlayer.currentPlaybackTime < 5 {
    self.musicPlayer.skipToPreviousItem()
} else {
    self.musicPlayer.skipToBeginning()
}

You can see that we use a new property here: currentPlaybackTime. This is the length of the song, in seconds, that has been played by our player. If the number of seconds played so far is less than 5, we call upon a new method: skipToPreviousItem(). This method, provided by the MediaPlayer framework, we jump back in the queue and start playing the song before this song. Currently, this will continue playing the beginning of this song.

However, what we can test is that if currentPlaybackTime is greater than or equal to 5, we can call skipToBeginning() in our musicPlayer and that will ask our musicPlayer to start playing from the beginning of our current song.

Build and run your app! It should be able to handle the skip and rewind buttons as you implemented! As you can see from this section, a lot of the methods we use to control playback are already implemented in the MediaPlayer framework. With a little research, you can continue to build upon all these features.

Implementing Album Artwork

Finally, let’s replace the ugly “A” SF Symbol with the album artwork of the current song playing. If you remember from earlier, our Song object has a property called artworkURL that contains the string of a URL containing the image of the album.

Normally, we’d have to write a lot of code to process the URL, gather the image data, and cache the image so we wouldn’t make redundant calls. I’d like to show you a nifty Swift package called [SDWebImageSwiftUI][6]. This package contains a framework built for SwiftUI that can help with image loading and contains features like async image loading, memory/disk caching, animated image playback and more. If you ever need to load images from a URL in your apps, I highly recommend this framework since it automatically handles image loading and caching, thereby speeding up your app.

Here’s how we can install it. We’ll be using Swift Package Manager to link this framework with our project. In Xcode, go to the Title Bar, and click on File > Swift Packages > Add Package Dependancy.

A popup will show up and ask you to enter a package repository URL. The URL we will be using is https://github.com/SDWebImage/SDWebImageSwiftUI. Enter this URL and press Next.

After some loading, Xcode will ask you which version to use. Don’t make any changes and simple press Next. Xcode will take some time to fetch the repository and create a package to add to your project. After a minute, Xcode will show you a popup confirming you to choose the packages and target. Make sure it looks like something below.

The important point to follow here is that MusicPlayer is selected as the target. Click Finish and Xcode will take you to the main project page that shows all your Swift Packages. This means that the package has successfully been added to your project and you can start using the framework!

Go to SearchView.swift and at the top of the file, type import SDWebImageSwiftUI. This will gives us access to all the objects and methods in this framework.

In our List object, locate the Image object that is currently coded as such:

Image(systemName: "rectangle.stack.fill")
    .resizable()
    .frame(width: 40, height: 40)
    .cornerRadius(5)
    .shadow(radius: 2)

Making sure that all its modifiers are still in place, replace it with the following

WebImage(url: URL(string: song.artworkURL.replacingOccurrences(of: "{w}", with: "80").replacingOccurrences(of: "{h}", with: "80")))
    .resizable()
    .frame(width: 40, height: 40)
    .cornerRadius(5)
    .shadow(radius: 2)

SDWebImageSwiftUI provides an object called WebImage. This is very similar to our Image object but has some really neat additions. For one, it takes an argument called url which when provided, automatically loads the image from this URL.

We create a URL object and pass the string song.artworkURL. Notice that once again, we perform string manipulation here. If you notice the URL, you’ll notice that it has two parameters: {h} and {w}. These parameters stand for height and width, respectively. This is provided by the Apple Music API on purpose because we can specify the height and width that is best suited for us. I chose a height and width of 80, since this is 2x the size of our WebImage so the image loaded will have a high quality. Therefore, we call replaceOccurences on both {w} and {h} and replace it with 80.

Everything else should remain the same! Make sure your code looks like the screenshot below:

Build and run your app! Now, when you search for a song, you can actually see the album cover in the search results. The best part too is that SDWebImageSwiftUI caches the image so if you quit the app and search for the same term again, you can notice that there is a huge speed increase in loading the image.

Our last step is to change the album artwork in PlayerView.swift. Unfortunately, this is not as simple. See, in SearchView.swift we had a list of Song objects, each containing a URL to the album cover. Since we didn’t pass this list along, our Player View can’t access this album artwork URL. Furthermore, while MediaPlayer is great at providing many built-in objects and methods, it does not contain an object with a link to a particular song’s album artwork.

In order to fix this, we need to rely on State and Binding, once again. Head to ContentView.swift and at the top of the file, underneath where you created your musicPlayer variable, add the following file:

@State private var currentSong = Song(id: "", name: "", artistName: "", artworkURL: "")

We are creating a new shared object called currentSong of type Song that will be updated whenever we click on a new song from our Search View.

On that note, go to SearchView.swift and add the following Binding property at the top of the file, below where you declared your musicPlayer variable.

@Binding var currentSong: Song

Scroll to the bottom of the file and you’ll find the Button object that currently sets the queue of the musicPlayer and tells it to start playing. Over here, we want to set currentSong to the song that was tapped upon.

Modify the action of the Button as shown.

Button(action: {
    self.currentSong = song
    self.musicPlayer.setQueue(with: [song.id])
    self.musicPlayer.play()
})

All we’re adding here is setting the currentSong to the song that was tapped on.

Let’s do the same for PlayerView.swift. Navigate to that file and at the very top, add import SDWebImageSwiftUI. This will let us use the WebImage object as we did above.

Next, add the Binding variable currentSong to the structure, right below where we declared the isPlaying variable.

@Binding var currentSong: Song

Last but not least, let’s replace our current Image object which holds the SF Symbol a.square with the following code. Notice that all the modifiers remain the same.

WebImage(url: URL(string: self.currentSong.artworkURL.replacingOccurrences(of: "{w}", with: "\(Int(geometry.size.width - 24) * 2)").replacingOccurrences(of: "{h}", with: "\(Int(geometry.size.width - 24) * 2)")))
    .resizable()
    .frame(width: geometry.size.width - 24, height: geometry.size.width - 24)
    .cornerRadius(20)
    .shadow(radius: 10)

Now just like before, we pass a URL to WebImage, only this URL is very lengthy. Just like before, we are replacing {w} and {h} with the width and height we want. It’s not as simple as before however, since our height and width is reliant on the width of the device. This is why we replace {w} and {h} with geometry.size.width - 24. The number is actually of type Floatso to convert it to an integer, we wrap it around in Int. Finally, we multiply both lengths by 2 to get a sharper image quality.

Last but not least, we still need to pass the currentSong from our Content View to both the Player View and Search View. Head over to ContentView.swift and make the following changes to where you declare both PlayerView() and SearchView().

PlayerView(musicPlayer: self.$musicPlayer, currentSong: self.$currentSong)
...
SearchView(musicPlayer: self.$musicPlayer, currentSong: self.$currentSong)

In these changes, we passing the currentSong that was created in ContentView.swift to the remaining views.

Congratulations! This is the end of this lengthy tutorial! Build and run your app and watch with delight as your app can make calls to the Apple Music API, parse JSON and display a list of songs, utilize a devices media player capabilities to play songs and control playback, and harness external libraries to load images from remote sources- all using SwiftUI, a technology that is only a year old!

Conclusion

If you’ve reached the end successfully, congratulations! This two part tutorial was not your average, intermediate tutorial. Despite its lengthy content, I hope you learned something new from all the technologies we’ve utilized. I’ve compiled a list of resources below that can help you if you choose to continue building upon this app! You can download the completed project on GitHub.

For reference, you can check out the following resource related to this tutorial:

Apple Documentation and Videos

• Apple Music API Documentation
• StoreKit Documentation
• MediaPlayer Documentation
• MusicKit on Android and the Web
• WWDC 2017 Video- Introduction MusicKit
• WWDC 2018 Video- MusicKit on the Web

More on the Technology

• Introducing JSON
• Working with JSON and Codable in Swift 5
• More on @State and @Binding in SwiftUI

As always, if you have any doubts, feel free to leave a comment below or reach me on Twitter @HeySaiK. I can’t wait to see what you’ll make with MusicKit!

At WWDC 2017, Apple announced MusicKit, a framework to help developers build apps that allow users to play Apple Music and their local music library. Unlike most frameworks like ARKit or CoreML, MusicKit cannot be added to your code with a simple import function. Rather, it’s a combination of the Apple Music API, the StoreKit framework, the MediaPlayer framework, and some other web-based technologies.

You may be wondering why is this framework so difficult to integrate into your applications compared to other API and frameworks. This is because MusicKit was not only built to work on iOS devices, but Android and Web applications as well. Just like CloudKit or Sign in with Apple, you can access a user’s Apple Music account through Android and Web apps. However, for the purpose of this tutorial, we will simply focus on building a music player for iOS using SwiftUI.

Editor’s note: This is a two part series of our MusicKit tutorials. This is the first part and the second one will come next week. If you are new to SwiftUI, you can check out this tutorial and this sample chapter of our Mastering SwiftUI book.

The MusicKit Demo App

In this tutorial series, we’ll be building a very simple music player that searches using the Apple Music API for a song, grabs the relevant details of the song like artist name and album cover, and plays it on the device with the help of MediaPlayer’s MPMusicPlayerController. We’ll also see how to use the MediaPlayer framework to play, rewind, and skip songs.

Before we can do this, we need to first communicate with the Apple Music service to see if a user has a valid Apple Music subscription. This requires us to create a MusicKit identifier and private key to sign your developer tokens using Certificates, Identifiers & Profiles in our Apple Developer Program account. Since this can be very daunting for a lot of iOS developers, a majority of this tutorial will focus on creating these keys and identifiers. The next part of this tutorial will focus more on the MediaPlayer framework.

Our final app will look like this:

This tutorial was made using Xcode 11.3 and Swift 5.1. An active Apple Music subscription will be needed in order to test the app out. You may also need an active Apple Developer account in order to create some of the required identifiers and keys. You will also need to run your app on a real device because as of this time, the Simulator does not support Apple Music playback. We will also be running some Python code in order to generate private keys for us. If you don’t have pip installed, don’t worry, I will explain how to later in the tutorial.

Creating a MusicKit Identifier

The first step in communicating with the Apple Music API is to have a valid MusicKit identifier in your Apple Developer account. This identifier will be linked to our app and will let the Apple Music service know that our app is valid to access its services. Since our identifier is linked to our app’s Bundle Identifier, let’s quickly create the Xcode project first.

Open Xcode and click on Create a new Xcode Project. Under iOS, make sure that Tabbed App is the option selected before clicking on the Next button.

Next, name the project to MusicPlayer but feel free to change it if you like. Make sure that your User Interface  is set to SwiftUI. Finally, take note of the Bundle Identifier as displayed below. For example, my bundle identifier is “com.appcoda.MusicPlayer”. But you should use your own.

Click on Next and choose where you would like to save your project. That’s it! You’ve created the project and just for reference, you should see your Bundle Identifier again on your Project Dashboard.

Now we can create our MusicKit Identifier. Head to your Apple Developer account. Once you login, you should be greeted with a screen that looks like this. Make sure you click on Certificates, Identifiers, & Profiles.

Once the page loads, click on Identifiers on the left side of the page. You’ll get a list of all the App IDs your Apple Developer account is associated with.

Now, let’s create a new MusicKit Identifier. Click on the blue plus button next to the Identifiers title. A new page will load titled Register a New Identifier. Make sure that you select the Music IDs option before pressing Continue.

Next, you’ll be asked to quickly fill out a description and identifier for your Music ID. You can see what I filled out below. For your identifier, make sure that it is in the following format- music.<YOUR-BUNDLE-ID>.

Click Continue and make sure all your details are correct, click on Register.

That’s it! If all worked out, you should see a list of all your Music IDs and one of them being the one you just created!

Next, we’ll create a private key that is associated with the identifier we just created. This key can be used to sign a secret developer token that allows us to communicate with MusicKit.

Creating a Private Key

Generally speaking, private keys are important because they allow you to access and authenticate communication with some app services — such as Push Notifications, MusicKit, and DeviceCheck. We’ll need to create a private key in order to sign something called a developer token. This token will be used when sending requests to the Apple Music API in order to let them know that this request is coming from a verified developer. You’ll use the private key to create the token in the form of a JSON web token (JWT). You can learn more about JWT here.

Still in the Certificates, Identifiers, and Profiles page, click on the Keys tab to the left side of the page. You’ll see a list of all the keys associated with your developer account. If you’ve ever used Push Notifications in your apps, you’ll find the key you used for that service.

Let’s create a new key for our app. Just like before, click on the plus button next to Keys. Name your key (I’m using “AppCoda MusicPlayer”) and select MusicKit from the list of capabilities shown below.

Before you select Continue, we’ll need to configure this capability. Click on Configure and you’ll be shown a page where you can associate this private key with the MusicKit Identifier we created earlier. Select the Music ID you created and then click Save.

You’ll go to the previous page where you can finally click on Continue. After a quick checkup to make sure everything has been entered correctly, click on Register.

If all goes well, you’ll see a page like the one above asking you to download your key. Make sure you download the key and save it in a safe place as you will not be able to download your key again. Also, make sure you take note of your Key ID as we’ll need it soon.

Now that we have the private key downloaded, we’ll need to use it to create the developer token in the JWT format. However, if you look at the file you downloaded, you’ll see that it’s in a .p8 file format and you probably can’t open the file. Luckily, there’s a way to view the private key from this file.

Open a new tab in your browser and head to https://filext.com/file-extension/P8. This website will read your .p8 file and display the private key to you. Drop your file into the placeholder and wait for a couple seconds.

You can see that the private key is given to you. I’ve hidden it from my screenshot above. You should also strive to keep this key as much of a secret since this is directly linked to your developer account. Copy the private key and save it. We’ll also need this in a bit.

Python Script

Now it comes to the fun part, but slightly complicated. You’ll need to create the developer token using a Python script. So, first download the script file here. In order for the script to work, you’ll need to install some Python packages. MacOS already comes with Python installed, so you don’t have to do any extra work to install Python. All you need to do is install those Python packages.

First, let’s download pip. If you’re not familiar with Python, pip is a very simple and easy-to-use package management system used to install and manage software packages written in Python. You can think of it as CocoaPods or Carthage for Python. Open Terminal. Type the following command:

sudo easy_install pip

After entering your password, pip will be installed. If you have pip installed already, you’ll probably see the following screen which means we can move to the next party.

Now we need two packages in order for our script to work: pyjwt and cryptography. Both are used to encrypt all the sensitive information and transform it into the JWT format. Just like before enter the following two commands in terminal:

sudo pip install pyjwt
sudo pip install cryptography

As long as you don’t get any errors (usually in red text), you should be in the clear. We’re almost done. Now we need to edit the script in order to populate it with 3 pieces of information.

1. The MusicKit private key (copied from the website which displayed our .p8 file)
2. The Key ID from the private key we created in the Apple Developer website
3. Your Apple ID Team ID

You can find your Team ID by going to https://developer.apple.com/account and clicking on the Membership tab to the left.

Now, open the script you downloaded earlier. You’ll be able to see where to enter your own details in the script.

Some notes about modifying the script:

• Copy and paste your MusicKit private key exactly as it’s displayed. This means there should be 4 lines in between the BEGIN PRIVATE KEY and END PRIVATE KEY (which should also be included).
• In case you get confused in all the strings we’ve seen, your Key ID is an alphanumeric 10 character string.

When you’re done entering all your information, save the script and head back to Terminal. Type the following command:

cd PATH/TO/WHERE/MUSICTOKEN.PY/IS/LOCATED

Now, for the magic. Type the command into Terminal:

python musictoken.py

If you’ve done everything right, you’ll see your token and a sample CURL request. Your Terminal screen will look like mine below.

If you face any errors, make sure you go back and see if you edited the script correctly. You may also want to make sure that you have the right Key ID, MusicKit private key, and Team ID. If none of these work, leave a comment below and I’ll be happy to help.

If your Terminal screen looks like mine, congratulations! You’ve successfully completed the hardest party of this tutorial. Copy your developer token and save it as we’ll be using it in the next part of the tutorial. For the rest of the tutorial, let’s go back to Swift and build the base layout of our app.

Creating the UI of the Music Player app

Open the Xcode project we created earlier in the tutorial. You can see that we already have our Tab View created for us in ContentView.swift. Let’s create a new SwiftUI view for our player view. Go to File > New > File and make sure you have SwiftUI View selected as your preset.

Save this file as PlayerView.swift. And, you should see the very basic SwiftUI View.

Let’s change this up. Command-click on the Text view and choose Embed in VStack. This will be the view that displays our song’s title. Let’s add another Text view inside the VStack. This can be the view that displays the artist’s name. Change the values of both Text structs to reflect the change.

VStack {
    Text("Song Title")
    Text("Artist Name")
}

In order to offer a clearer distinction between both Text views, I’ll change the font and add some spacing between both views. This is for a purely design-based purpose. You can modify these values to whatever you see fit. Here is the code:

VStack(spacing: 8) {
    Text("Song Title")
        .font(Font.system(.title).bold())
    Text("Artist Name")
        .font(.system(.headline))
}

That’s much better! The users of our app can clearly distinguish between our song’s title and its artist. Now let’s add an Image view so we can show the album cover to our users! Before we do that, let’s copy all the code we have so far and paste it into a GeometryReader. This is a container view that defines its content as a function of its own size and coordinate space. It helps with resizing views for different device sizes.

Click on the Plus button in the top right corner of Xcode and search for Geometry Reader.

Select this and drag it to paste the code about where we declare our VStack. Now copy the VStack struct and paste it inside the GeometryReader’s placeholder. While our preview will still look the same, our code should now look like this.

GeometryReader { geometry in
    VStack(spacing: 8) {
        Text("Song Title")
            .font(Font.system(.title).bold())
        Text("Artist Name")
            .font(.system(.headline))
    }
}

Now we can add an Image view that will function as a place where we can display the album cover. Modify your code to look like this.

GeometryReader { geometry in
    // 1
    VStack(spacing: 24) {
        // 2
        Image(systemName: "a.square")
            .resizable() // 3
            .frame(width: geometry.size.width - 24, height: geometry.size.width - 24) // 4
            .cornerRadius(20)
            .shadow(radius: 10)

        VStack(spacing: 8) {
            Text("Song Title")
                .font(Font.system(.title).bold())
            Text("Artist Name")
                .font(.system(.headline))
        }
    }
}

1. First, I take the existing VStack and wrap it inside another VStack. This offers some spacing in between the Image view and the Text views.
2. Next, I created the Image view. Since the app has no data right now, I’m using an SF Symbol as a placeholder image.
3. In order to resize the Image, it’s important to add the .resizable() function.
4. Finally, I define the frame of my Image view. geometry.size.width is the width of the device. You can think of it like a SwiftUI version of self.view.frame.size.width. I also subtract 24 in order to create some margins.

Your preview should look a little like this now.

Now we need to create our play, rewind, and skip buttons. Click on the plus button like before and drag a button right below the VStack where we define our song’s information.

Let’s modify this button so it functions as our rewind button. Change the Button view code to look like this.

Button(action: {
    // 1
    print("Rewind")
}) {
    // 2
    ZStack {
        // 3
        Circle()
            .frame(width: 80, height: 80)
            .accentColor(.pink)
            .shadow(radius: 10)
        Image(systemName: "backward.fill")
            .foregroundColor(.white)
            .font(.system(.title))
    }
}

1. Currently since we don’t have access to the MediaPlayer framework, we’ll just print a statement when the button is tapped.
2. Next we define the look of our button. I decided to use a ZStack so we can use an SF Symbol as an image and use the Circle view as its background.
3. I create the Circle view first and give it its frame, color, and shadow for design purposes. I then use an SF Symbol to provide an image for the button.

We need two more of these buttons next to each other. Command and click on the Button view and click on Embed in HStack. Copy the button code and paste it inside the HStack struct two more times.

Obviously we don’t want our player to look like this so let’s modify the HStack code.

HStack(spacing: 40) {
    Button(action: {
        print("Rewind")
    }) {
        ZStack {
            Circle()
                .frame(width: 80, height: 80)
                .accentColor(.pink)
                .shadow(radius: 10)
            Image(systemName: "backward.fill")
                .foregroundColor(.white)
                .font(.system(.title))
        }
    }

    Button(action: {
        print("Pause")
    }) {
        ZStack {
            Circle()
                .frame(width: 80, height: 80)
                .accentColor(.pink)
                .shadow(radius: 10)
            Image(systemName: "pause.fill")
                .foregroundColor(.white)
                .font(.system(.title))
        }
    }

    Button(action: {
        print("Skip")
    }) {
        ZStack {
            Circle()
                .frame(width: 80, height: 80)
                .accentColor(.pink)
                .shadow(radius: 10)
            Image(systemName: "forward.fill")
                .foregroundColor(.white)
                .font(.system(.title))
        }
    }
}

In the code above, I’m simply adding a spacing of 40 between my buttons. All of them are similar to the button’s I’ve created earlier. The only difference is that each button prints a different statement regarding its function and has a different image.

Your player view should look like this now:

Next, let’s build the Search View.

Search View

Just like before, create a new SwiftUI view and name this file SearchView.swift. You’ll be greeted with the familiar template view.

Our search view will be very simple. It just needs a TextField view for searching songs and a List view for displaying the results. Meanwhile, we also need to populate some fake data since the app hasn’t connected with the Apple Music API. At the top of the file, right before you declare body, type the following:

@State private var searchText = ""
let songs = ["Blinding Lights", "That Way", "This Is Me"]

var body: some View {
    Text("Hello, World!")
}

The searchText variable is the binding variable we’ll use to connect our TextField view with. We define the songs array to populate our list with some temporary data.

Let’s build the TextField view first. Remove the Text view and replace it with the following code:

VStack {
    TextField("Search Songs", text: $searchText, onCommit: {
        print(self.searchText)
    })
    .textFieldStyle(RoundedBorderTextFieldStyle())
    .padding(.horizontal, 16)
    .accentColor(.pink)
}

I’ve created the TextField view with a placeholder text of “Search Songs” and attached any text type into this View to the searchText variable. The onCommit method is called when the user pressed Return on their keyboard. In the next part of the tutorial, we’ll implement our search functionality within this method.

Next, I’ll create the List view and populate it with the data from above. Underneath the TextField, paste the following code.

List {
    // 1
    ForEach(songs, id:\.self) { songTitle in
        // 2
        HStack {
            // 3
            Image(systemName: "rectangle.stack.fill")
                .resizable()
                .frame(width: 40, height: 40)
                .cornerRadius(5)
                .shadow(radius: 2)

            // 4
            VStack(alignment: .leading) {
                Text(songTitle)
                    .font(.headline)
                Text("Artist Name")
                    .font(.caption)
                    .foregroundColor(.secondary)
            }
            Spacer()
            // 5
            Button(action: {
                print("Playing \(songTitle)")
            }) {
                Image(systemName: "play.fill")
                    .foregroundColor(.pink)
            }
        }
    }
}
.accentColor(.pink)

If you’ve worked with the List view or the ForEach structure before, this should look familiar. If not, I’ll walk you through the code.

1. Inside my List view, I initialize the ForEach structure. This makes it easy to create multiple copies of views that I won’t have to explicitly code. As my data, I pass the songs array we created earlier. I also pass each element in that array as a variable called songTitle. This is so we can easily populate our cells without having to worry about array indexes.
2. Next, to emulate the look of a UITableViewCell, I create an HStack. I’ll be placing the album cover, song title, song artist, and add button within each cell inside the HStack.
3. Next is the Image view. As before, I’m using an SF Symbol as a placeholder image.
4. Next comes our Text views. I place them inside a VStack to conserve space and display multiple Text views in the same place. You’ll notice that the string value of our first Text view is the songTitle variable we created earlier.
5. Finally, I add a Play button which indicates to the user that our music player will play this song. Currently, it only prints to the console. In the next part of the tutorial, we’ll manage adding songs to the queue inside this method.

That’s all! Take a look at your Canvas and make sure it looks something like this:

Putting it all together

We’re almost done! All that’s left is going back to ContentView.swift and modifying the inside of our TabView to make sure it displays the two views we have created. Delete all the code inside TabView and replace it with the following.

TabView(selection: $selection) {
    PlayerView()
        .tag(0)
        .tabItem {
            VStack {
                Image(systemName: "music.note")
                Text("Player")
            }
        }
    SearchView()
        .tag(1)
        .tabItem {
            VStack {
                Image(systemName: "magnifyingglass")
                Text("Search")
            }
        }
}
.accentColor(.pink)

In the above code, I’m calling both PlayerView() and SearchView() as the two views in our TabView. As such, we’ll only have two tabs. I’ve also created the look of those tabs with a simple Image and Text.

By now you must have noticed that I’ve been using Color.pink inside all my buttons and .accentColors. This provides the app with a unique, music-themed design. You’re free to change the colors, fonts, and shapes to however you wish.

What’s Next

Congratulations! You’ve done a lot in this tutorial by venturing out of Swift and into further technologies like Python, Web Token, Encryption, and API handling. This was a lot but if you’ve made it to the end, congrats! The hardest part about using MusicKit is generating your keys and developer tokens. You can download the final project here.

In the next part of this tutorial series, we’ll complete our app. I’ll show you how to access the Apple Music API and make web requests to the API. Furthermore, I’ll teach you how to use MediaPlayer to play music and control playback from your device. If you have any doubts, feel free to leave a comment. Stay tuned!

2 years ago, at WWDC 2017, Apple released the Vision framework, an amazing, intuitive framework that would make it easy for developers to add computer vision to their apps. Everything from text detection to facial detection to barcode scanners to integration with Core ML was covered in this framework.

This year, at WWDC 2019, Apple released several more new features to this framework that really push the field of computer vision. That’s what we’ll be looking at in this tutorial.

What We’ll be Building in this Tutorial

For many years now, Snapchat has reigned as the popular social media app among teens. With its simple UI and great AR features, high schoolers around the world love to place cat/dog filters themselves. Let’s flip the script!

In this tutorial, we’ll be building Snapcat, the Snapchat for Cats. Using Vision’s new animal detector, we’ll be able to detect cats, place the human filter on them, and take pictures of them. After taking pictures of our cats, we’ll want to scan their business cards. Using a brand new framework, VisionKit, we’ll be able to scan their business cards just like the default Notes app on iOS.

That’s not all! If you remember 2 years ago my tutorial on using Vision for text detection, I ended by saying that even though you could detect text, you would still need to integrate the code with a Core ML model to recognize each character. Well finally, Apple has released a new class under Vision to recognize the text it detects. We’ll use this new class to grab the information from the scanned cards and assign it to our cat. Let’s get started!

The project in this tutorial was built using Swift 5.1 in Xcode 11 Beta 7 running on macOS Catalina. If you face any errors when go through the tutorial, try to update your Xcode to the latest version or leave a comment below.

Getting the Starter Project

To begin, download the starter project here. Build and run on your device. What we have is the iconic Snapchat camera view. To create this, I added a ARSCNView, or an SceneView that is capable of displaying AR content, and overplayed it with a button.

When the shutter button is pressed, the app takes a snapshot of the current frame of this scene view and passes it along the navigation stack to the Cat Profile View where it’s loaded the profile image. In this view, we have empty fields for the data: Name, Number, and Email. This is because we haven’t scanned the business card yet! Clicking on the button has no action yet because this is where we’ll implement the document scanning program via VisionKit.

Image Recognition – Detecting a Cat Using VNRecognizeAnimalsRequest

Our first step is to detect cats in our camera view. Head over to CameraViewController and at the top of the file, type import Vision. This imports the Vision framework into this file, giving us access to all the classes and functions.

Now we want our app to automatically place the human filter (“the human emoji”) on our cat with no input from us whatsoever. This means that we need to scan the frames in our camera view every few milliseconds. To do this, modify your code to look like below:

@IBOutlet var previewView: ARSCNView!
var timer: Timer?

override func viewWillAppear(_ animated: Bool) {
    super.viewWillAppear(animated)
    let configuration = ARWorldTrackingConfiguration()
    previewView.session.run(configuration)
    timer = Timer.scheduledTimer(timeInterval: 0.5, target: self, selector: #selector(self.detectCat), userInfo: nil, repeats: true)

}

We create an instance of timer and define it in our viewWillAppear function. We schedule the timer to perform the function detectCat() in the interval of every half a second.

However, you’ll experience an error displayed because we have not created the function called detectCat. This is a simple fix. Here is the viewWillAppear function. Insert the code in the CameraViewController class:

@objc func detectCat() {
    print("Detected Cat!")
}

Let’s build and run the project so far. The moment our CameraViewController is initialized, we should see the text “Detected Cat!” printed to the console every half second.

Now here comes to the fun part where we’ll use the new VNRecognizeAnimalsRequest, a built-in class that lets you detect cats and dogs.

Modify the detectCat() function to look like this:

@objc func detectCat() {
    guard let currentFrameBuffer = self.previewView.session.currentFrame?.capturedImage else { return }
    let image = CIImage(cvPixelBuffer: currentFrameBuffer)
    let detectAnimalRequest = VNRecognizeAnimalsRequest { (request, error) in
        DispatchQueue.main.async {
            if let results = request.results?.first as? VNRecognizedObjectObservation {
                let cats = result.labels.filter({$0.identifier == "Cat"})
                for cat in cats {
                    print("Found a cat!!")
                }
            }
        }
    }

    DispatchQueue.global().async {
        try? VNImageRequestHandler(ciImage: image).perform([detectAnimalRequest])
    }
}

There’s quite a bit going on here but I’ll go through it line by line. First, we implement a guard function that checks to make sure that the current frame in our Preview View has an image. If this image can be captured, we create a special CIImage using the pixel buffer of that frame’s image.

Here’s some terminology! The CIImage class we’re using is a representation for an image as a CoreImage. This type of images are great when you want to analyze the pixels in them. A pixel buffer stores an image data in the main memory.

Before explaining the next few lines of code, it’ll help to understand how Vision works in case you’re unfamiliar. Basically, there are 3 steps to implement Vision in your app, which are:

1. Requests – Requests are when you request the framework to detect something for you.
2. Handlers – Handlers are when you want the framework to perform something after the request is made or “handle” the request.
3. Observations – Observations are what you want to do with the data provided with you.

So first, we create a VNRecognizeAnimalRequest to detect animals. This is a new VNImageRequest that was introduced in iOS 13 specifically for animal detection.

Next, within the handler of this request, we get the result and filter the labels in our first result. Why are we doing this? This is because currently, the VNRecognizeAnimalRequest can detect both cats and dogs. For our use, we only want cats which is why we define an array called cats where the identifier is equal to “Cats”. Finally, for each cat observation, we print that we found a cat to the console. This is just to see if Vision is detecting cats.

Remember, we’ve only defined the request. Now we need to ask Vision to perform the request. After defining the request, we ask VNImageRequestHandler to perform the request on our CIImage we defined earlier. A VNImageRequestHandler is basically a specific type of VNRequest that is used to process images using one or more image analysis requests.

Build and run the project on a device. I don’t have a cat so I’ll be using some images from online. If Vision correctly detects the cat, then we should be able to see it printed to the console.

It works!! But how should we let users know that the app detects a cat? It’s time to code again.

At the top of the class where we initialized the timer object, type the line: var rectangleView = UIView(). This initializes a UIView which we can modify to be a box. We’ll place this box around the coordinates where our Vision function can detect the cat.

Inside the detectCat() method, insert self.rectangleView.removeFromSuperview() at the beginning. Then modify the for loop to look like this:

for cat in cats {
    self.rectangleView = UIView(frame: CGRect(x: result.boundingBox.minX * self.previewView.frame.width, y: result.boundingBox.minY * self.previewView.frame.height, width: result.boundingBox.width * self.previewView.frame.width, height: result.boundingBox.height * self.previewView.frame.height))

    self.rectangleView.backgroundColor = .clear
    self.rectangleView.layer.borderColor = UIColor.red.cgColor
    self.rectangleView.layer.borderWidth = 3
                        self.previewView.insertSubview(self.rectangleView, at: 0)
}

Here’s what we’re doing. We are defining our rectangleView to have the bounds of the result’s bounding box. However, this is something worth mentioning.

In Vision, the size of the result’s bounding box is smaller than the size of your previewView. This is because when Vision analyzes the image, it scales down the image to speed up computer vision processes. That’s why when we create the bounds for our rectangleView, we have the multiply it by the constant to scale it up.

After defining the frame, all we do is set the view to have a transparent background and a border of width 3 and color red. We add this to our previewView. Now, you may be wondering why we included the line of code self.rectangleView.removeFromSuperview() at the beginning of the function. Remember, our Vision code is running every half a second. If we don’t remove the rectangleView, we’ll have a lot of boxes on our view.

Ok! Build and run the app! Are you seeing the boxes?

Yes! It’s working. Now we have one last thing to do: place a human emoji near the cat. This is really simple. Here’s what we do.

Ath the top of the class, underneath where we define our rectangleView, type the line: var humanLabel = UILabel().

Inside the for loop of our detectCat() function, we need to add this label to our view. Here’s how to do that.

for cat in cats {
    self.rectangleView = UIView(frame: CGRect(x: result.boundingBox.minX * self.previewView.frame.width, y: result.boundingBox.minY * self.previewView.frame.height, width: result.boundingBox.width * self.previewView.frame.width, height: result.boundingBox.height * self.previewView.frame.height))

    self.humanLabel.text = "👦"
    self.humanLabel.font = UIFont.systemFont(ofSize: 70)
    self.humanLabel.frame = CGRect(x: 0, y: 0, width: self.rectangleView.frame.width, height: self.rectangleView.frame.height)

    self.rectangleView.addSubview(self.humanLabel)
    self.rectangleView.backgroundColor = .clear
    self.previewView.insertSubview(self.rectangleView, at: 0)
} 

This is really easy to follow. We still have the same frame for our rectangleView. Now, we set the text of the label to be an emoji followed by the font size. Since we want the labels to cover the entire box, we set its width and height to be the same as the rectangleView. Finally, we removed the code that would create a border and instead add the label to our box and add the box to the previewView.

Build and run the app. Is the code working? Can you see the human emoji on the cat?

It works! We’ve got a human emoji placed on the cat. Now, of course, it’s not perfect because Vision only detects a cat object, not the facial features of the cat. Hopefully Apple will add that soon and we can further refine the filter.

Scanning the Business Card Using VNDocumentCameraViewController

Whew! We made it through the first part. From here on, it gets a little easier. When you snap the picture of your cat, you automatically get pushed to the Cat Profile view where you can see the image you’ve taken, the details of the cat, and a button that says “Scan For Details“. We’ll be pulling up the new document scanner code when this button is tapped on.

Navigate to CatProfileViewController.swift. At the top of the file, underneath import UIKit, type import VisionKit. The difference between Vision and VisionKit is that while Vision lets your perform computer vision analyses on an image, VisionKit is a small framework that lets your app use the system’s document scanner.

Now within the scanDocument() function, all you need to add are three lines of code.

@IBAction func scanDocument(_ sender: Any) {
    let documentCameraViewController = VNDocumentCameraViewController()
    documentCameraViewController.delegate = self
    self.present(documentCameraViewController, animated: true, completion: nil)
}

The VNDocumentCameraViewController() is a special type of view controller created by Apple which is used to scan the documents. We set the delegate of that controller and present the view controller.

Next, you need to implement the VNDocumentCameraViewControllerDelegate in the class. Change the line where we define out CatProfileViewController class to look like this.

class CatProfileViewController: UIViewController, VNDocumentCameraViewControllerDelegate {
    ....
}

Build and run the app. After taking a picture, when you press “Scan for Details”, the device’s document scanner should pop up.

Looks like it’s scanning! However, when we press “Save”, nothing happens. This is because we haven’t implemented any delegate methods yet. Underneath, the scanDocument() function, type the following.

func documentCameraViewController(_ controller: VNDocumentCameraViewController, didFinishWith scan: VNDocumentCameraScan) {
    let image = scan.imageOfPage(at: 0)
    self.catImageView.image = image
    controller.dismiss(animated: true)
}

The documentCameraViewController(didFinishWith:) is a method that runs when the Save button is clicked. We take the first scan, make it an image, and set the catImageView to display this image. We’re doing this to make sure that the scanner is working well.

Build and run the app. When you have to scan for details, scan over any business card and when you click “Save”, it should dismiss the document scanner and set the image to the catImageView.

Now that the app is able to detect the business card, let’s further implement it to extract the textual data from the card.

Text Recognition Using VNRecognizeTextRequest

With the new API, VNRecognizeTextRequest, introduced in iOS 13, it’s pretty easy to finds and recognizes text in an image.

Again, first import VisionKit in the file because the new API is bundled in the framework. As explained earlier, this will bring over the Vision API’s. Next, before the viewDidLoad() function, type the following:

var textRecognitionRequest = VNRecognizeTextRequest()
var recognizedText = "" 

The VNRecognizeTextRequest is just like the VNRecognizeAnimalsRequest class that we’ve used earlier. This request tells Vision to run text recognition on whatever image we pass to it. Later, when Vision detects text in a image, we’ll assign it to the variable recognizedText.

Head over to the viewDidLoad() function and update the code like below:

 override func viewDidLoad() {
super.viewDidLoad()
    self.navigationController?.setNavigationBarHidden(false, animated: true)
    self.title = "New Cat"
    catImageView.image = catImage

    textRecognitionRequest = VNRecognizeTextRequest(completionHandler: { (request, error) in
        if let results = request.results, !results.isEmpty {
            if let requestResults = request.results as? [VNRecognizedTextObservation] {
                self.recognizedText = ""
                for observation in requestResults {
                    guard let candidiate = observation.topCandidates(1).first else { return }
                      self.recognizedText += candidiate.string
                    self.recognizedText += "\n"
                }
                self.catDetailsTextView.text = self.recognizedText
            }
        }
    })
}

In the code, we initiate a VNRecognizeTextRequest. Vision returns the result of this request in a VNRecognizedTextObservation object. This type of observation contains information about both the location and content of text and glyphs that Vision recognized in the input image.

For every item in the requestResults array, we choose the first topCandidates. What is this? Well for every observation Vision makes, it outputs a list of potential candidates for the detected text. In some cases, it may be useful to consider all the possible text. For our case, we only care about the most accurate prediction.

Finally, we take the candidate.string and add it to our recognizedText. After going through every observation in the requestResults array, we set the text of the catDetailsTextView to our recognizedText.

Your code should look a little like the image below:

You can build and run the app now but nothing will happen. Why? This is because we still haven’t defined a handler to perform the request. This can be accomplished in a couple lines of code. Head back to the documentCameraViewController(didFinishWith:) function and modify it like this:

func documentCameraViewController(_ controller: VNDocumentCameraViewController, didFinishWith scan: VNDocumentCameraScan) {
    let image = scan.imageOfPage(at: 0)
    let handler = VNImageRequestHandler(cgImage: image.cgImage!, options: [:])
    do {
        try handler.perform([textRecognitionRequest])
    } catch {
        print(error)
    }
    controller.dismiss(animated: true)
}

The code above should look familiar as we used it in the first section of the tutorial. We define a handler of type VNImageRequestHandler and input the cgImage of our scan.

A VNImageRequestHandler is an object that processes one or more image analysis requests pertaining to a single image. By specifying the image, we can begin executing the Vision request. In the do block, we ask the handler to perform the Vision request we defined earlier. x

Build and run the app. Is it working?

It seems to be working quite well! We’re able to gather all the text from the scanned card and place it onto the text view.

Fine Tuning the Text Recognition

Now in my image above, the text recognition worked great. However, I’ll try to help you fine tune our VNRecognizeTextRequest in case it didn’t work so well for you.

At the end of our viewDidLoad() function, type the following:

textRecognitionRequest.recognitionLevel = .accurate
textRecognitionRequest.usesLanguageCorrection = false
textRecognitionRequest.customWords = ["@gmail.com", "@outlook.com", "@yahoo.com", "@icloud.com"] 

These are some of the values that the VNRecognizeTextRequest has. You can modify them to fine tune your application. Let’s go through these.

1. .recognitionLevel: You have two options between these: .fast and .accurate. When you want the request to recognize text, you can choose between recognizing faster or more accurately. Apple advices to use .accurate because even though it take slightly more time, it works well for recognizing texts in custom font shapes and size. The advantage with .fast is that it takes up less memory on the device.
2. .usesLanguageCorrection: Here’s how Vision Text Recognition Request works. After grabbing the text from each observation, it doesn’t just output that text to you. It goes through another layer of Natural Language to change any misspelled words. Think if it like a spell and grammar check. By changing this Boolean value, you can enable this on or off. When should you set this to true? If your application is being used to recognize text from a book or a document, then you should enable this setting. However, in our scenario, we’re using it to detect names, numbers, and email addresses. If we enable this setting, it’s possible that the request may think the number 1 is the lowercase letter l. Or another example is the last name He which Vision may correct to Hey or Hi.
3. .customWords: This value is an array that can be used for specific words. In our scenario, we’re detecting emails which generally end in @email.com. While this isn’t a word on its own, we wouldn’t want Vision predicting these characters to be something else. By offering custom words, the text recognition request analyzes the image a second time to see if any of those words appear so it can recognize them properly.

Build and run the app. Is there any improvement? Did the recognition system get better or worse? By fine tuning these parameters, you can utilize Vision’s text recognition system to be the best in your app.

Conclusion

In this tutorial, as you can see, it is so easy to take advantage of all the new API’s in Vision to perform object recognition. You learned how we can use the brand new animal detector to recognize cats and dogs in an image. You should also understand how the new framework, VisionKit, makes scanning documents a breeze. Finally, the coveted text recognition APIs became available in Vision. With easy-to-use API’s and lots of parameters for fine tuning the system, you can easily implement text recognition in your apps.

However, this isn’t all that was announced at WWDC 2019. Vision also introduced API’s to classify images into categories, analyze saliency, and detect human rectangles. I’ve attached some great links if you want to continue learning more about Vision.

• Vision Documentation
• WWDC 2019 Session: Understanding Images in Vision Framework
• WWDC 2019 Session: Text Recognition in Vision Framework
• AppCoda Tutorial: Using Vision Framework for Text Detection in iOS 11

I’d love to know what you’re building with Vision! Leave a comment below if you have a question or want to share what you’re working on. You can download the full tutorial here. Thanks for reading! Catch you in the next article!

One of the most innovative inventions Apple has come up with in the past year is its True Depth camera. An ode to hardware and software engineers, the True Depth camera is what powers its secure facial recognition system, FaceID. As developers, the True Depth camera opens up a world of possibilities for us, especially in the field of face-base interactions.

Before we begin this ARKit tutorial, let me quickly brief you on the different parts of the camera. Like most iPhone/iPad front cameras, the True Depth camera comes with a microphone, a 7 megapixel camera, an ambient light sensor, a proximity sensor, and a speaker. What makes the True Depth camera itself is the addition of a dot projector, flood illuminator, and infrared camera.

The dot projector projects more than 30,000 invisible dots onto your face to build a local map (you’ll see this later in the tutorial). The infrared camera reads the dot pattern, captures an infrared image, then sends the data to the Secure Enclave in the A12 Bionic chip to confirm a match. Finally, the flood illuminator allowed invisible infrared light to identify your face even when it’s dark.

These parts come together to create some magical experiences like Animojis and Memojis. Special effects that require a 3D model of the user’s face and head can rely on the True Depth Camera.

Introduction to the Demo Project

I believe that it’s important for developers to learn how to utilize the True Depth camera so they can perform face tracking and create amazing face-based experiences for users. In this tutorial, I will show you how we can use the 30,000 dots to recognize different facial movements using ARFaceTrackingConfiguration, that comes with the ARKit framework.

The final result will look like this:

Let’s get started!

You will need to run this project on either an iPhone X, XS, XR, or iPad Pro (3rd gen). This is because these are the only devices which have the True Depth camera. We will also be using Swift 5 and Xcode 10.2.

Editor’s Note: If you’re new to ARKit, you can refer to our ARKit tutorials.

Creating a ARKit Demo for Face Tracking

First, open Xcode and create a new Xcode project. Under templates, make sure to choose Augmented Reality App under iOS.

Next, set the name of your project. I simply named mine True Depth. Make sure the language is set to Swift and Content Technology to SceneKit.

Head over to Main.storyboard. There should be a single view with an ARSCNView already connected to an outlet in your code.

What we have to do is really simple. All we need to do is add a UIView and a UILabel inside that view. This label will inform the user of the face expressions they are making.

Drag and drop a UIView into the ARSCNView. Now, let’s set the constraints. Set the width to 240pt and height to 120pt. Set the left and bottom constraints to 20pt.

For design purpose, let’s set the alpha of the view to 0.8. Now, drag a UILabel into the view you just added. Set the constraints to 8 points all around as shown below.

Finally, set the alignment of the label to centralized. Your final storyboard should look like this.

Now, let’s set the IBOutlets to our ViewController.swift file. Switch to the Assistant editor. Control and click on the UIView and UILabel and drag it over to ViewController.swift to create the IBOutlets.

You should create two outlets: faceLabel and labelView.

Creating a Face Mesh

Let’s clean up the code a little bit. Because we chose the Augmented Reality App as our template, there’s some code which we don’t need. Change your viewDidLoad function to this:

override func viewDidLoad() {
    super.viewDidLoad()

    // 1 
    labelView.layer.cornerRadius = 10

    sceneView.delegate = self
    sceneView.showsStatistics = true

    // 2
    guard ARFaceTrackingConfiguration.isSupported else {
        fatalError("Face tracking is not supported on this device")
    }
}

With the template, our code loads a 3D scene. However, we don’t need this scene, so we delete it. At this point, you can delete the art.scnassets folder in the project navigator. Finally, we add two pieces of code to our viewDidLoad method.

1. First, we round the corners of the labelView. This is more of a design choice.
2. Next, we check to see if the device supports the ARFaceTrackingConfiguration. This is the AR tracking setting we’ll be using to create a face mesh. If we don’t check to see if a device supports this, our app will crash. If the device does not support the configuration, then we will present an error.

Next, we’ll change one line in our viewWillAppear function. Change the constant configuration to ARFaceTrackingConfiguration(). Your code should look like this now.

Next, we need to add the ARSCNViewDelegate methods. Add the following code below // MARK: - ARSCNViewDelegate.

func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? {
    let faceMesh = ARSCNFaceGeometry(device: sceneView.device!)
    let node = SCNNode(geometry: faceMesh)
    node.geometry?.firstMaterial?.fillMode = .lines
    return node
}

This code runs when the ARSCNView is rendered. First, we create a face geometry of the sceneView and set it to the constant faceMesh. Then, we assign this geometry to an SCNNode. Finally, we set the material of the node. For most 3D objects, the material is usually the color or texture of a 3D object.

For the face mesh, you can use two materials- either a fill material or a lines material. I prefer the lines which is why I set fillMode = .lines, but you can use what you prefer. Your code should look like this now.

If we run the app, you should see something like this.

Updating the Face Mesh

You may notice that the mesh does not update when you change your facial features (blinking, smiling, yawning, etc.). This is because we need to add the renderer(_didUpdate:) under the renderer(_nodeFor) method.

func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {
    if let faceAnchor = anchor as? ARFaceAnchor, let faceGeometry = node.geometry as? ARSCNFaceGeometry {
        faceGeometry.update(from: faceAnchor.geometry)
    }
}

This code runs every time the sceneView updates. First, we define a faceAnchor as the anchor for the face it detects in the sceneView. The achor is the information about the pose, topology, and expression of a face detected in the face-tracking AR session. We also define the constant faceGeometry which is a topology of the face detected. Using these two constants we update the faceGeometry every time.

Run the code again. Now, you’ll see the mesh updating every time you change your facial features, all running at 60 fps.

Analyzing the features

First, let’s create a variable at the top of the file.

var analysis = ""

Next, type the following function at the end of the file.

func expression(anchor: ARFaceAnchor) {
    // 1
    let smileLeft = anchor.blendShapes[.mouthSmileLeft]
    let smileRight = anchor.blendShapes[.mouthSmileRight]
    let cheekPuff = anchor.blendShapes[.cheekPuff]
    let tongue = anchor.blendShapes[.tongueOut]
    self.analysis = ""

    // 2    
    if ((smileLeft?.decimalValue ?? 0.0) + (smileRight?.decimalValue ?? 0.0)) > 0.9 {
        self.analysis += "You are smiling. "
    }

    if cheekPuff?.decimalValue ?? 0.0 > 0.1 {
        self.analysis += "Your cheeks are puffed. "
    }

    if tongue?.decimalValue ?? 0.0 > 0.1 {
        self.analysis += "Don't stick your tongue out! "
    }
}

The above function takes an ARFaceAnchor as a parameter.

1. The blendShapes are a dictionary of named coefficients representing the detected facial expression in terms of the movement of specific facial features. Apple provides over 50 coefficients which detect various different facial features. For our purpose we’re using only 4: mouthSmileLeft, mouthSmileRight, cheekPuff, and tongueOut.
2. We take the coefficients and check the probability of the face performing these facial features. For detecting a smile, we add the probabilities of both the right and left side of the mouth. I found that the 0.9 for the smile and 0.1 for the cheek and tongue work best.

We take the possible values and add the text to the analysis string.

Now that we have our function created, let’s update our renderer(_didUpdate:) method.

func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {
    if let faceAnchor = anchor as? ARFaceAnchor, let faceGeometry = node.geometry as? ARSCNFaceGeometry {
        faceGeometry.update(from: faceAnchor.geometry)
        expression(anchor: faceAnchor)
        
        DispatchQueue.main.async {
            self.faceLabel.text = self.analysis
        }
        
    }
}

We run the expression method every time the sceneView is updated. Since the function is setting the analysis string, we can finally set the text of the faceLabel to the analysis string.

Now, we’re all done coding! Run the code and you should get the same result as we saw in the beginning.

Conclusion

There is a lot of potential behind developing face-based experiences using ARKit. Games and apps can utilize the True Depth camera for a variety of purposes. One of my favorite apps is Hawkeye Access, a browser you can control using your eyes.

For more information on the True Depth camera, you can check out Apple’s video Face Tracking with ARKit. You can download the final project here.

Last year, Apple debuted a new framework called NaturalLanguage. This was the next step in making text-based machine learning much more accessible to developers all around. This year, the progress has not stopped as Apple has continued to make advancements in this framework.

In this tutorial, we’ll see what are some of the new API’s in this framework, as well as, what else is possible with the use of NaturalLanguage.

You’ll need to use Xcode 11 and Swift 5.1 in order to follow the tutorial and build the sample app. At the time of writing, these softwares are still in beta, so some functionalities may not be the same after the official release.

I have enabled the Mac device in order to test Project Catalyst, while working on this sample app. You’ll find some of the screenshots below are a Mac app.

What’s Natural Language Processing?

What exactly is Natural Language Processing (NLP)? Simply put, this framework gives apps the ability to analyze natural language text and understand parts of it. This framework can perform a variety of tasks on text by assigning tag schemes to the text.

So, what are tag schemes? Well, basically tag schemes are the constants used to identify the pieces of information we want from the text. You can think of them as a set of tasks we ask a tagger to apply to the text. Some of the most common tag schemes we ask the tagger to look for are the language, name type, lemma, etc.

All NLP tasks can be split into 2 sections: Text Classification and Word Tagging. At WWDC 2019, Apple announced improvements in both these sections of NaturalLanguage. We’ll go through them one by one to see what’s new!

The Starter Project

Before you continue reading the tutorial, please first download the starter project.

In our starter project, you can see that we have an app called Text+. This is a tabbed application that is used to separate the new API’s we’ll work with. At the end of our project, we should have an app that runs Sentiment Analysis in one tab and Word Embedding in the other tab. I’ll explain what these are in more depth later on. Here’s what you’ll be building.

Text Classification

Text Classification is the process of assigning a label to a group of text, such as a sentence or a paragraph or even a whole document. These labels can be anything you choose: either a topic label, sentiment label, or any label that help you classify it.

New in NaturalLanguage this year is the new built-in API for Sentiment Analysis. Sentiment Analysis is the task of classifying a block of text by it’s mood. From a score of -1.0 to 1.0, we can determine how positive or negative a group of text is.

In the demo application, choose the SuggestionViewController.swift file, you should see that we have a function called analyzeText(). This function will be called when the analyzeButton is tapped.

The user is free to type any message in the text field. What we want to do is to perform sentiment analysis on that message and change the color according of the button accordingly:

1. green if the message is positive,
2. red if the message is negative,
3. Or blue if the message is neutral.

Let’s start to implement the change. Underneath where we declare our IBOutlets, let’s declare our tagger.

import UIKit
import NaturalLanguage

class SentimentViewController: UIViewController {
    @IBOutlet var analyzeButton: UIButton!
    @IBOutlet var messageTextField: UITextField!
    let tagger = NLTagger(tagSchemes: [.sentimentScore])

    ...
}

Our tagger is an NLTagger where we ask it to observe the scheme of sentimentScore. This means that when we assign this tagger to our text, it will run the process of looking for a sentiment score.

Our next step is to add the logic to our analyzeText() function. Here’s how you do it:

@IBAction func analyzeText() {
    tagger.string = messageTextField.text
    let (sentiment, _) = tagger.tag(at: messageTextField.text!.startIndex, unit: .paragraph, scheme: .sentimentScore)
    print(sentiment!.rawValue)
}

If you’ve worked with NaturalLanguage before, this small block of code should be self-explanatory. If not, here’s what it means.

First, we assign the text in the messageTextField to our tagger by setting it in the string property. Next, we call the tag method of the tagger. This method will find a tag based on the given text and the specified scheme. Here the scheme we used is .sentimentScore. This scheme will score the text as positive, negative, or neutral based on its sentiment polarity.

The tag method also requires you to specify the linguistic unit. Since the user may type a paragraph of text, we specify the unit to .paragraph.

The call will return two values: the tag and the range in which our sentiment is detected. Since we don’t need it, we use _ in the code above. Finally, we print the value of sentiment.

You can now run the code and see the score printed to the console.

You can see that for my sentence, “I am feeling very happy that I was able to code this!”, I got a score of 0.6 which is mostly positive!

We’re done with the NLP part! Next, I challenge you to change the background color of analyzeButton in respect to the score we get. If the score is greater than 0, this means that it is positive so we’d like the background color to be green. If the score is less than 0, this means that it is negative so we’d like the background color to be red. Finally, if the score is 0, this means that it is neutral so we’d like the background color to be blue.

Were you able to do it? If not, no worries. Here’s how it can be done. Add the following lines of code in our analyzeText() function.

let score = Double(sentiment!.rawValue)!
if score < 0 {
    self.analyzeButton.backgroundColor = .systemRed
} else if score > 0 {
    self.analyzeButton.backgroundColor = .systemGreen
} else {
    self.analyzeButton.backgroundColor = .systemBlue
}

This code is pretty easy. We transform the raw value of our sentiment constant into a type Double. Then, using an if-else statement, we change the background color of our button.

If possible, try to use .system colors in your apps from now on because they’re are universal within Apple’s OS and can automatically be adjusted for features like Dark Mode and Accessibility without you having to do anything.

Congratulations! With just a few lines of code, you were able to code a full Sentiment Analysis feature into your app. Previously, you would have to rely on a CoreML model but with NaturalLanguage, this just became much more easier! Next, let’s see what’s new with Word Tagging.

Word Tagging

Word Tagging is slightly different than Text Classification. In this process, given that we have a sequence of words (or tokens as it is commonly referred to in NLP), we want to assign a label to every single token. This could be assigning each token its respective part of speech, named entity recognition, or any other tagging system that helps you assign it to each token.

One important task of NLP which falls under Word Tagging is Word Embedding. Word embedding is a mapping of objects into a vector representation. A vector is nothing more than a continuous sequence of numbers. Well, what does this mean and why is it important? By using this method of mapping objects (or tokens in our case), we can find a way to quantitatively organize a group of objects. So when you plot these vectors, you can find that similar objects are clustered together. This helps when we want to build something that can also display objects similar to it. Apart from words, embedding can also be used for images, phrases, and more!

In the second part of our app, we’ll be building a suggestion system for someone going shopping with the help of word embedding. Here’s how we can achieve it. Looking at our storyboard, you can see that we have a text field for entering the item we’re looking for, a button to suggest items for us, and a label which will show us those suggestions. Since all of our UI is hooked up, all we need to do is add the following lines of code to our suggest() function.

@IBAction func suggest(_ sender: Any) {
    //1
    suggestionLabel.text = "You may be interested in:\n"
    let embedding = NLEmbedding.wordEmbedding(for: .english)

    //2
    embedding?.enumerateNeighbors(for: itemTextField.text!.lowercased(), maximumCount: 5) { (string, distance) -> Bool in
        //3
        print("\(string) - \(distance)")
        suggestionLabel.text! += (string.capitalized + "\n")
        return true
    }
}

Let me explain the code above line by line:

1. First, we want to reset the suggestionLabel so the previous suggestions won’t be there. You use an NLEmbedding to find similar strings. The framework provides some built-in word embeddings that we can use on-the-fly. Here, by calling its wordEmbedding method with our preferred language, we can use the returned NLEmbedding for finding similar words in English.
2. With the NLEmbedding, we call the enumerateNeighbors method to find similar words for the input text. I have configured it to return only 5 of the closest neighbors but you can change that number.
3. Finally, I print the neighboring string and its distance from our input string to the console. This distance indicates the similarity of the two words. The smaller the distance, the higher is the similarity.

Build and run the project! You should see everything work flawlessly!

Now, you may notice that this isn’t the ideal experience since word embeddings only display words that are close to it, not necessarily the similarity of objects. For example, if I ask a word embedder to show me the closest words for “Meat”, one of the words would be “Vegetarian”. While this is a close word, it definitely is not something a shopper would be looking for. For our app, a Recommender System would be more apt. This also requires machine learning and will be covered in an upcoming tutorial on CreateML.

Where else can word embedding be used? In the Photos application, when you search for “sea”, it also shows you pictures of the beach, surfboard, and other sea-related pictures. This is possible through word embedding.

Another application of word embedding is with something called Fuzzy Searching. Has it every happened to you when you searched for something, either a song or a book, and you misspelled maybe a few letters but you still got the right result. This is because Fuzzy Searching uses Word Embedding to calculate the distance between your wrong input and the right input in the database which allows it to show the right phrase.

What else is new

Sentiment Analysis and Word Embedding were the top 2 API’s in this year’s update of NaturalLanguage. On top of that, there are 2 more features that were released as well. While going into them would make the tutorial too long, I’ll quickly cover these new features.

Custom Word Embeddings

Earlier, we used a general word embedding model that is built into all of Apple’s operating systems. However, sometimes, we may want a word embedding model that is based on specifically one domain, such as financial words or medical words. Apple has thought of that and NaturalLanguage also supports the use of custom word embeddings.

When we create recommender systems in Create ML, under the hood, it builds a custom word embedding using the data provided. However, if you’d like to control the algorithms, then Apple also allows us to do that as well. You can easily build your own word embedder using CreateML

import CreateML

let vectors = ["Object A": [0.112, 0.324, -2.24, ...],
                "Object B": [0.112, 0.324, -2.24, ...],
                "Object C": [0.112, 0.324, -2.24, ...],
                  ...
              ]
let embedding = try MLWordEmbedding(dictionary: vectors)
try embedding.write(to: url)

You can obtain the vectors for these words through a custom neural network using either TensorFlow or PyTorch. This is way beyond the scope of the tutorial but if you’re interested, I highly suggest you take a look on the web because this is an active research area in NLP and can be quite interesting.

Text Catalog

We briefly mentioned how you can create your own word tagger with MLWordTagger. However, it would take a lot of efforts to create the word tagging model.

New this year is the addition of MLGazetteer. A gazetteer is a text catalog, or a dictionary, filled with names and labels. CreateML then takes this gazetteer and transforms it into a tagger that can be used in your app. For example, if you wanted a tagger that would tag electronic devices by its type, you would use a file that would look like this:

["smartphone": ["iPhone XS", "Samsung S10", "Google Pixel 3a", ...],
"laptop": ["MacBook Pro", "Surface Laptop", "Chromebook", ...],
"smartwatch": ["Apple Watch", "Samsung Galaxy Watch", "Fitbit Versa", ...],
...
]

Ideally, in this JSON file, you would have thousands of entities. Then, you’d use CreateML to create your gazetteer like this:

import CreateML

let entities = ["smartphone": ["iPhone XS", "Samsung S10", "Google Pixel 3a", ...],
"laptop": ["MacBook Pro", "Surface Laptop", "Chromebook", ...],
"smartwatch": ["Apple Watch", "Samsung Galaxy Watch", "Fitbit Versa", ...],
...
]
let gazetteer = try MLGazetteer(dictionary: entities)
try gazetteer.write(to: url)

Then you would use it with the NaturalLanguage framework as such:

import NaturalLanguage

let gazetteer = try! MLGazetteer(contentsOf: url)
let tagger = NLTagger(tagSchemes: [.nameTypeOrLexicalClass])
tagger.setGazetteers([gazetteer], for: .nameTypeOrLexicalClass)

If you’d like to see a second part of this tutorial with the above features explained more in depth, leave a comment below!

Conclusion

As shown, you can see how the field of Natural Language Processing is quickly expanding and Apple is doing its best to make these high level technologies much more accessible for developers. I suggest you take a look at some of the resources below such as the Apple’s documentation and the WWDC 2019 sessions given about this framework.

• Natural Language Documentation
• WWDC 2019 – Advancements in Natural Language Framework

With powerful tools like Sentiment Analysis and Word Embedding, you can create a whole range of apps in all domains that can leverage the power of machine learning. If you have any questions, feel free to ask in the comments below.

For reference, you can download the completed project here.

WWDC 2019 was one of the more exciting keynotes in terms of advancements in developer tools. One of the biggest and the best announcements was the release of SwiftUI. SwiftUI is a brand new framework that allows you to design and developer user interfaces with way less code and in a declarative way.

Unlike UIKit which was commonly used conjoined with storyboards, SwiftUI is completely based on code. However, the syntax is very easy to understand and can quickly be previewed with Automatic Preview.

Since SwiftUI is built with Swift, it allows you to create the same complexity of apps with much less code. What’s even more is that the use of SwiftUI automatically enables your app to take advantage of features like Dynamic Type, Dark Mode, Localization, and Accessibility. Furthermore, SwiftUI is available on all platforms including macOS, iOS, iPadOS, watchOS, and tvOS. So now, your UI code can be synchronized across all platforms giving you more time to focus on the minor platform-specific code.

Editor’s note: This tutorial has been updated for Xcode 11.4 and Swift 5.2. If you want to dive deeper into SwiftUI, you can check out our Mastering SwiftUI book.

About this tutorial

It’s important that developers learn early how to use SwiftUI because Apple will eventually migrate much of their focus to this framework. In this tutorial, we’ll look at the basics of SwiftUI and explore how to create navigation views, images, texts, and lists by building a simple contact list that shows all our tutorial team members. When a member is clicked, the app proceeds to the detail view displaying their picture along with a short bio about them. Let’s get started!

This tutorial requires you to be running Xcode 11 (or up) and we will be using Swift 5 in this tutorial. While an advanced knowledge of Swift is not necessary to follow the tutorial, it’s recommended that you understand the basics.

Setting Up Your Project with SwiftUI

Let’s start from scratch so you can see how to start to run a SwiftUI app immediately. First, open Xcode and click on Create new Xcode project. Under iOS, choose Single View App. Name your app and fill out the text fields. However, at the bottom, make sure that the Use SwiftUI option is checked. If you do not check the option, Xcode will generate the storyboard file

Now Xcode will automatically create a file for you called ContentView.swift and the amazing part is that it will show you a live preview of your code to the right hand side as shown below.

If you don’t see the preview, you will need to hit the Resume button in the preview canvas. It’ll take some time to build the project. Just be patient.

Now let’s start seeing how we can modify these files to create our app.

Building the List View

To build the list view, there are three parts to this. The first part is the creation of the rows in the list. You may recognize the design to be similar to a UITableView. To do this, we have to create a ContactRow. The second part is connecting the data we need to our list. I have the data already coded and it takes just a few modifications to connect our list to the data. The final part is simply adding a Navigation Bar and embedding our list in a Navigation View. This is pretty simple. Let’s see how we implemented all these in SwiftUI.

Creating the Tutor List

First, let’s create the list view for displaying a list of all the tutorial team members including their profile photos and description. Let’s see how this can be done.

As you can see in the generated code, we already have a Text component with the value set to “Hello World”. In the code editor, change the value of the code to “Simon Ng”.

struct ContentView: View {
    var body: some View {
        Text("Simon Ng")
    }
}

If all works, you should see your canvas on the right update automatically. That’s the power of instant preview that we’ve all anticipated for so long.

Let’s add another Text view to our app. This will be the short description of the member. To add a new UI element to our app, press the + button in the top right corner. A new window will appear with a list of different views. Drag the view titled Text and drop it underneath our initial Text view as shown below.

Notice the code on the left:

struct ContentView: View {
    var body: some View {
        VStack {
            Text("Simon Ng")
            Text("Placeholder")
        }
    }
}

You can see that a new Text view was added underneath our Simon Ng text view. What’s different is that it seems to have wrapped it in something called a VStack. A VStack is short for vertical stack and it is the replacement of Auto Layout in SwiftUI. If you’ve developed for watchOS before, you know that there are no constraints, rather everything is placed into groups. With a vertical stack, all of our views will be arranged vertically.

Now, change the text of “Placeholder” to “Founder of AppCoda”.

Next, let’s add an image to the left of this text. Since we want to arrange a view horizontally to the existing views, we need to wrap the VStack in an HStack. To do this, CMD+Click on the VStack in your code, and then choose Embed in HStack. Take a look at it below:

Your code should now look like this:

struct ContentView: View {
    var body: some View {
        HStack {
            VStack {
                Text("Simon Ng")
                Text("Founder of AppCoda")
            }
        }
    }
}

There are no distinctive changes in our automatic preview but let’s add an image now. Change your code to look like this:

struct ContentView: View {
    var body: some View {
        HStack {
            Image(systemName: "photo")
            VStack {
                Text("Simon Ng")
                Text("Founder of AppCoda")
            }
        }
    }
}

Starting from iOS 13, Apple introduces a new feature called SFSymbols. Designed by Apple, SF Symbols is a set of over 1,500 symbols you can use in your app. Since they are able to integrate seamlessly with the San Francisco system font, the symbols automatically ensure optical vertical alignment with text for all weights and sizes. Since we don’t have the images of our tutors yet, we use a placeholder image here.

Now, let’s focus on some minor design issues. Since we want to emulate the look and feel of a UITableRow, let’s align the text to the left (i.e. leading). To do that, CMD+Click on the VStack and click on Inspect. Select the left alignment icon as shown below:

You’ll see the code change to the following. And, you’ll also see your live preview change to reflect the new changes.

VStack(alignment: .leading) {
    ...
}

Now since our second text view is a headline, let’s change the font to reflect this. Just like before, CMD+Click on the “Founder of AppCoda” text view in the live preview and select Inspect. Change the font to “Subheadline” and watch both the live preview and your code change.

Let’s change the color too and set it to “Gray”. Your code should now look like this:

struct ContentView: View {
    var body: some View {
        HStack {
            Image(systemName: "photo")
            VStack(alignment: .leading) {
                Text("Simon Ng")
                Text("Founder of AppCoda")
                    .font(.subheadline)
                    .color(.gray)
            }
        }
    }
}

Now that we’re done designing the sample row, here comes to the magic part. Watch how easy it is to create a list. CMD+Click on our HStack and click on Embed in List. Voila! Watch how your code will automatically change and our canvas will reflect 5 beautiful new rows each showing Simon Ng as the team member.

Also, be sure to note how the List was created in the code. By removing the HStack and replacing it with a reiterating List, we were able to create a table view. Now think about all the time and lines of code you saved avoiding all the UITableViewDataSource, UITableViewDelegate, Auto Layout, Implementation for Dark Mode, etc. This alone shows the extent and power of SwiftUI. However, we’re far from done. Let’s add some real data to our new list.

Connecting our data to the list

The data we need is a list of the tutorial team members and their bios along with a folder of all their images. You can download the files you need here. You should find 2 files named Tutor.swift and Tutor.xcassets.

Once downloaded, import both the Swift file and the asset folder into your Xcode project. To import them, simply drag them to the project navigator.

In Tutor.swift, we declare a struct called Tutor and have it conform to the Identifiable protocol. You’ll see why this is important later. We also define its variables as id, name, headline, bio, and imageName. Finally, we include some test data that will be used in our app. In Tutor.xcassets, we have images of all our team members.

Go back to ContentView.swift and modify your code to look like this:

struct ContentView: View {
    //1
    var tutors: [Tutor] = []

    var body: some View {
        List(0..<5) { item in
            Image(systemName: "photo")
            VStack(alignment: .leading) {
                Text("Simon Ng")
                Text("Founder of AppCoda")
                    .font(.subheadline)
                    .color(.gray)
            }
        }
    }
}

#if DEBUG
struct ContentView_Previews : PreviewProvider {
    static var previews: some View {
        //2
        ContentView(tutors: testData)
    }
}
#endif

What we’re doing here is quite simple:

1. We’re defining a new variable named tutors which is an empty array of the structure Tutor.
2. Since we’re defining a new variable to our structure ContentView, we need to change the ContentView_Previews as well to reflect this change. We set the parameter tutors to our testData.

There won’t be any change in our live preview because we haven’t used our test data. To display the test data, modify your code like this:

struct ContentView: View {
    var tutors: [Tutor] = []

    var body: some View {
        List(tutors) { tutor in
            Image(tutor.imageName)
            VStack(alignment: .leading) {
                Text(tutor.name)
                Text(tutor.headline)
                    .font(.subheadline)
                    .color(.gray)
            }
        }
    }
}

Here, we make sure the ContentView to use the tutors list for displaying the data on screen.

Ta-da! Look at how your live preview changes in the canvas.

The images are in square shape. We’d like them to have more of a rounded and circular look. Let’s see how we can make a rounded-corner image. In the top right corner, press the + button and click the second tab. This should show a list of layout modifiers that you can add to the views.

Search for “Corner Radius”, drag it over to our live preview and drop it on the image. You should see the code and the live preview change to something like this.

However, a corner radius of 3 is a little too small to notice. So, change it to 40. This way, you’ll achieve a nice and circular profile picture.

The cell and the list are all done now! Give yourself a pat on the back for accomplishing this. What we need to do next is present a detail view when a user taps on a cell. Let’s start building the navigation.

Building the Navigation

A navigation view wraps the view we already have in a navigation bar and navigation controller. Assuming you’ve used Storyboard before, you should know that it’s pretty easy to embed a view in a navigation interface. All you need to do is just a few clicks.

For SwiftUI, wrapping the List view in a NavigationView is also very easy. All you need to do is to change your code like this:

...
var body : some View {
    NavigationView {
        List(tutors) { tutor in 
            ...
        }
    }
}
...

You just need to wrap the List code in a NavigationView wrapper. By default, the navigation bar doesn’t have a title. Your preview should move the list down leaving a very large gap in the middle. This is because we haven’t set the navigation bar’s title. To fix that, we can set the title by adding the following line of code (i.e. .navigationBarTitle):

...
var body : some View {
    NavigationView {
        List(tutors) { tutor in 
            ...
        }
        .navigationBarTitle(Text("Tutors"))
    }
}
...

Your screen should now look similar to this:

Next, let’s set up the navigation link. A NavigationLink presents a destination which is a view that presents on the navigation stack. Just like how we wrapped the List in a NavigationView, we need to wrap the content of the List with a NavigationLink as shown below:

...
var body : some View {
    NavigationView {
        List(tutors) { tutor in 
            NavigationLink(destination: Text(tutor.name)) {
                Image(tutor.imageName)
                VStack(alignment: .leading) {
                    Text(tutor.name)
                    Text(tutor.headline)
                        .font(.subheadline)
                        .foregroundColor(.gray)
                }
            }
        }
        .navigationBarTitle(Text("Tutors"))
    }
}
...

We simply shows the name of the team member in the detail view. Now’s the time to test it out.

In the current preview mode, you can’t interact with the view. Normally when you click on the automatic preview, it just highlights the code. In order to test and interact with the UI, you need to press the play button at the bottom right corner.

The view will go dark and you may need to wait for a couple of seconds for the whole simulator to load before you can actually interact with the views.

Once it finishes loading, you should be able to click on a cell and it will navigate to a new view in the stack that displays the name of the selected cell.

Before moving onto the implementation of the detailed view, let me show you a nifty trick that will help making your code more legible. CMD+Click the NavigationLink and choose “Extract Subview”.

Boom! You can see that the entire code in NavigationLink has been created into a brand new struct that makes it very legible. Rename ExtractedView to TutorCell.

You may now get an error in TutorCell. This is because we don’t have a tutor parameter to pass in this new structure. It’s very simple to fix the error. Add a new constant in your TutorCell struct as such:

struct TutorCell: View {
    let tutor: Tutor
    var body: some View {
        ...
    }
}

And, in the ContentView, add the missing parameter by changing the line to:

...
List(tutors) { tutor in 
    TutorCell(tutor: tutor)
}.navigationBarTitle(Text("Tutors"))
...

That’s it! We have our list and cells all well designed and laid out! Next, we’re going to build a detail view that will show all the information of the tutor.

Building the Detail View

Let’s create a new file by going to File > New > File. Under iOS, select SwiftUI View and name this file TutorDetail.

You can see in the automatic preview that our basic view has been created. Let’s change this. First, click on the + button and drop an image above the Text view already built in. Set the name of the image to “Simon Ng”. Simon’s picture should show up. Now modify your code to look like below:

struct TutorDetail: View {
    var body: some View {
        //1
        VStack {
            //2
            Image("Simon Ng")
                 .clipShape(Circle())
                .overlay(
                    Circle().stroke(Color.orange, lineWidth: 4)
                )
                .shadow(radius: 10)
            //3
            Text("Simon Ng")
                .font(.title)
        }
    }
}

You should be able to follow the code but in case you can’t, don’t worry. Here’s what it’s basically saying:

1. First, we wrap all our views in a vertical stack. This is crucial to the layout of the design we’ll be taking.
2. Next, we take that image of Simon and spice it up. First, we set the clips of the image to be in the shape of a circle. Rather than setting its cornerRadius, this is much more effective because it can adapt to different image sizes. We add an overlay of a circle with a white border which provides a beautiful orange border. Finally, we add a light shadow to provide some depth.
3. Our last line of code sets the font of the name of the tutor to the the title font.

We also need to add two more text views: headline and bio. Drag in two text views below the tutor name text view and let’s edit them:

struct TutorDetail: View {
    var body: some View {
        VStack {
            Image("Simon Ng")
                 .clipShape(Circle())
                .overlay(
                    Circle().stroke(Color.orange, lineWidth: 4)
                )
                .shadow(radius: 10)
            Text("Simon Ng")
                .font(.title)
            Text("Founder of AppCoda")
            Text("Founder of AppCoda. Author of multiple iOS programming books including Beginning iOS 12 Programming with Swift and Intermediate iOS 12 Programming with Swift. iOS Developer and Blogger.")
        }
    }
}

The good news is that we have our text views present. The bad news is it looks bad and doesn’t really show the difference between a headline and a biographical description. Furthermore, the bio text view doesn’t display the whole text. Let’s fix these issues one by one.

Update the code like this:

struct TutorDetail: View {    
    var body: some View {
        VStack {
            Image("Simon Ng")
                 .clipShape(Circle())
                .overlay(
                    Circle().stroke(Color.orange, lineWidth: 4)
                )
                .shadow(radius: 10)
            Text("Simon Ng")
                .font(.title)
            //1
            Text("Founder of AppCoda")
                .font(.subheadline)
            //2
            Text("Founder of AppCoda. Author of multiple iOS programming books including Beginning iOS 12 Programming with Swift and Intermediate iOS 12 Programming with Swift. iOS Developer and Blogger.")
                .font(.headline)
                .multilineTextAlignment(.center)

        }
    }
}

1. First we set the “Founder of AppCoda” to be a font weight of subheadline
2. Similarly, we set the bio text view to the font weight of headline. We also center the text with the line .multilineTextAlignment(.center).

Let’s fix the next problem. We need to display the entire text of the bio text view. This can be easily done by adding a new line of code:

...
Text("Founder of AppCoda. Author of multiple iOS programming books including Beginning iOS 12 Programming with Swift and Intermediate iOS 12 Programming with Swift. iOS Developer and Blogger.")
        .font(.headline)
        .multilineTextAlignment(.center)
        .lineLimit(50)
...

Everything looks good. There’s one last design change I want to make. The headline and the bio text views are rather too close to each other. I’d like to have some space in between these two Text views. Also, I’d like to add some paddings to all the views such that they are not hugging the edges of the device. Make sure you change the code like this:

struct TutorDetail: View {
    var body: some View {
        VStack {
            Image("Simon Ng")
                 .clipShape(Circle())
                .overlay(
                    Circle().stroke(Color.orange, lineWidth: 4)
                )
                .shadow(radius: 10)
            Text("Simon Ng")
                .font(.title)
            Text("Founder of AppCoda")
                .font(.subheadline)
            //1
            Divider()

            Text("Founder of AppCoda. Author of multiple iOS programming books including Beginning iOS 12 Programming with Swift and Intermediate iOS 12 Programming with Swift. iOS Developer and Blogger.")
                .font(.headline)
                .multilineTextAlignment(.center)
                .lineLimit(50)
        //2
        }.padding()
    }
}

We’ve made a couple of changes here:

1. Adding a divider is just as simple as calling Divider().
2. To add padding to the entire vertical stack, we just have to call .padding() at the end of the VStack declaration.

That’s all! Congrats! You finally completed the detail view. What’s left is connecting our list to the detail view. This can be quite simple.

Passing Data

To pass data, we need to declare some parameters in our TutorDetail struct. Before declaring the variable body, add the following variables:

var name: String
var headline: String
var bio: String
var body: some View {
    ...
}

These are the parameters we’ll pass from our ContentView. Replace the text occurrences with these parameters:

...
var body: some View {
    VStack {
        // 1
        Image(name)
            .clipShape(Circle())
               .overlay(
                   Circle().stroke(Color.orange, lineWidth: 4)
            )
               .shadow(radius: 10)
        //2
        Text(name)
            .font(.title)
        //3
        Text(headline)
            .font(.subheadline)
        Divider()
        //4
        Text(bio)
            .font(.headline)
            .multilineTextAlignment(.center)
            .lineLimit(50)
        //5
    }.padding().navigationBarTitle(Text(name), displayMode: .inline)
}
...

1. We replace the name of the image with our variable name
2. We also replace the name of the tutor with our variable name
3. We replace our headline text with our variable headline
4. Finally, we also replace our long paragraph of text with out variable bio
5. We also add a new line of code that will set the title of the navigation bar to the name of our tutor.

Last but not least, we need to add the missing parameters to our struct TutorDetail_Previews.

#if DEBUG
struct TutorDetail_Previews : PreviewProvider {
    static var previews: some View {
        TutorDetail(name: "Simon Ng", headline: "Founder of AppCoda", bio: "Founder of AppCoda. Author of multiple iOS programming books including Beginning iOS 12 Programming with Swift and Intermediate iOS 12 Programming with Swift. iOS Developer and Blogger.")
    }
}    
#endif

In the above code, we add the missing parameters and fill in the information with what we had earlier.

You may be wondering what’s up with the #if DEBUG/#endif statements. These lines of code mean that whatever code is wrapped within these commands, will only be shown in the preview for debugging purposes. In your final app, it won’t be there.

Resume your automatic preview. Nothing should change since we still have the same information. But make sure that you are able to preview TutorDetail before moving on to the next step.

Our final step is to link this view to our list. Switch over to the ContentView.swift file. All you need to do is change one line of code in the TutorCell struct. Change the NavigationButton code to the following:

...
var body: some View {
    return NavigationButton(destination: TutorDetail(name: tutor.name, headline: tutor.headline, bio: tutor.bio)) {
        ...
    }
}
...

Instead of presenting a Text view with the tutor name, we need to change the destination to that of TutorDetail while filling in the appropriate details. Your code should now look like this:

Click the play button on the live canvas and interact with the view. If all works well, it should work exactly as expected.

Simply choose one of the member records:

And then you should see the member details in the detail view.

Conclusion

This was a pretty huge tutorial but it presents the basics of SwiftUI. You should be comfortable to build simple apps now like a to-do list. I suggest you take a look at some of the resources below such as the Apple’s documentation and the WWDC 2019 sessions given about this framework.

• SwiftUI Documentation
• SwiftUI Tutorials
• Introducing SwiftUI: Building Your First App
• SwiftUI Essentials

This framework is the future of Apple development so it’s great to get that head start early on. Remember, if you’re confused about the code, try to interact with the automatic preview and see if you can make UI changes directly to see how the code is built. If you have any questions, feel free to ask in the comments below.

For reference, you can download the completed project here. If you want to dive deeper into SwiftUI, check out our next tutorial on SwiftUI button.

In October 2018, Apple announced the brand new iPad Pro and the all-new Apple Pencil 2.0. Unlike the previous generation of the Apple Pencil, this utensil offers developers some extra fun APIs to play around with in order to enhance their app’s functionality and UX. In this tutorial, I will show you how to make your app support the Apple Pencil 2.

Getting Started

We will be building an app called Canvas, where users can look at hilarious invention ideas from the parody account Bored Elon Musk everytime they double tap their Apple Pencil. First, open Xcode and select “New Project”. Choose “Single View App” and name your project to whatever name you like.

Before we begin, we need to do something important. Since the Apple Pencil is supported only on the iPad, we need to make sure that our project is set to iPad only but not Universal.

Setting up the User Interface

First, we will design the user interface of the app. Navigate to Main.storyboard and drag a label into the view controller, set it’s font size to 30, style to bold, and lines to 0.

Next, we have to set the constraints. Click on the Align button on the bottom right and checkmark “Horizontally in Container” and “Vertically in Container”.

We are almost done. Next, drag in a button and set its title to “Next Idea!” and font to “Semibold 25”.

Finally, click on the Align button. Check both “Horizontally in Container” and “Vertically in Container”. For “Vertically in Container”, set its value to 100. This way the button will not be placed right below the label. Also, set the alpha of the button to 0 so it’s invisible by default. I will explain why we did that later.

Let’s connect our UI elements to our code. Create two outlets in ViewController.swift and connect it to the UI elements.

I have two outlets titled ideaLabel and ideaButton. The ideaLabel will display our invention idea. In the event that the user does not have an Apple Pencil or has turned off Apple Pencil support, we will display the ideaButton so they can still use the app.

That’s the basics of our UI. Let’s jump to the coding part!

Coding the Support for Apple Pencil 2

First, we declare an array to hold all of our invention ideas. Insert the following code in the viewDidLoad method:

let ideas = ["Email app that anonymously pings all your coworkers after New Year’s", "Self-imposed ads reminding you of your to-do list", "Anti-bacterial gel dispensing doorknobs.", "Vending machines that can break a $20", "Routers that work."]

I’ve added a few ideas but you’re free to add as many as you want!

Next, we have to change the text of the label when the view loads up. We will randomly select an idea and put it on the label. This can be done quite simply by a single line of code. Update the viewDidLoad method like this:

override func viewDidLoad() {
    super.viewDidLoad()
    ideaLabel.text = (ideas.randomElement()!)
}

This sets the text of our ideaLabel to a random string from our array.

Now let’s run the app to have a test. If all goes well, the label should be changing every time you open the app.

Finally it comes the exciting part! We are now going to interact with the Apple Pencil to capture the double-tap action. In order to make it work, all you need to do is set up the Pencil Interaction delegate. In the viewDidLoad method, insert the following lines of code:

let interaction = UIPencilInteraction()
interaction.delegate = self
view.addInteraction(interaction)

Basically, we create a constant interaction of type UIPencilInteraction and set its delegate to self. Then we add this interaction to the view. However, as soon as you insert the code, you’ll notice an error in Xcode. This is because the ViewController class is not conformed to the UIPencilInteractionDelegate. To fix that, let’s adopt the delegate in the ViewController class.

If we run our app, you will notice that nothing changes. This is because we still have not written the code for the double-tap action. Now, create the pencilInteractionDidTap method in the ViewController class:

func pencilInteractionDidTap(_ interaction: UIPencilInteraction) {
    switch UIPencilInteraction.preferredTapAction {
    case .ignore:
        ideaButton.alpha = 1
    case .showColorPalette:
        ideaLabel.text = (ideas.randomElement()!)
    case .switchEraser:
        ideaLabel.text = (ideas.randomElement()!)
    case .switchPrevious:
        ideaLabel.text = (ideas.randomElement()!)
    default:
        break
    }
}

When an Apple Pencil 2 is connected to an iPad Pro, the user can choose what they would like the Apple Pencil to do including:

1. Ignore the tap
2. Show the color palette
3. Switch to an eraser
4. Switch to the last used tool

These rules were made in the thought that the Apple Pencil would be used mostly for drawing apps. However, you are allowed to tell the Apple Pencil to perform a lot more. That said, it’s important that you let your user know what the tap would be doing.

In the code above, unless the Apple Pencil’s configuration is set to ignore, we change the text of ideaLabel when the app captures the tap action. If it is set to ignore, we can deduce that the user won’t be using the Apple Pencil as much. In this case, we will make the ideaButton visible by setting its alpha to 1.

If you run the code, everything should work as expected. The only function that doesn’t work is the when you tap on the Next Idea! button. This is because we haven’t linked it to an IBAction. Just link them up and then the button will work.

After the changes, run the app to test again and everything should work flawlessly!

It’s hard to show the functionality of an app using screenshots. I shot a video to show you how the app works in action. Don’t forget to take a look!

Conclusion

The Apple Pencil is truly a fascinating accessory for iPad Pro users. For developers, it’s much more than a simple stylus. While this tutorial is primitive, the objective is to show you how to add Apple Pencil support to your iPad apps. As you can see, it’s pretty easy to do that. What to keep in mind when developing for the Apple Pencil and iPad Pro is to adopt the UIPencilInteractionDelegate.

I can’t wait to see what you’ll come up with this technology. If you want to learn more about developing for iPad Pro and Apple Pencil, I suggest you take a look at these videos:

• Designing for iPad Pro and Apple Pencil
• Bringing Your Apps to the New iPad Pro

For reference, you can check out the full Xcode project on GitHub.

If you’ve been following Apple’s announcements from the past year, you know that they are heavily invested in machine learning. Ever since they introduced Core ML last year at WWDC 2017, there are tons of apps which have sprung up which harness the power of machine learning.

However, one challenge developers always faced was how to create the models? Luckily, Apple solved our question last winter when they announced the acquisition on Turi Create from GraphLab. Turi Create is Apple’s tool which can help developers simplify the creation of their own custom models. With Turi Create, you can build your own custom machine learning models.

A Quick Introduction to Turi Create

If you’ve been following the other machine learning tutorials, you’re probably wondering, “Didn’t Apple announced Create ML this year? What are the advantages to Turi Create over CreateML?”

While Create ML is a great tool for people just getting started with ML, it is severely limited in terms of usage. With Create ML, you are limited to text or image data. While this does constitute for a majority of projects, it can be rendered useless for slightly more complex ML applications (like Style Transfer!).

With Turi Create, you can create all the same Core ML models as you can with Create ML but more! Since Turi Create is much more complex than Create ML, it is heavily integrated with other ML tools like Keras and TensorFlow. In our tutorial on CreateML, you saw the types of Core ML models we could make with Create ML. Here are the types of algorithms you can make with Turi Create:

• Recommender systems
• Image classification
• Image similarity
• Object detection
• Activity classifier
• Text classifier

You can see that this list consists of classifiers and regressors which can be accomplished with either Create ML, or mostly Turi Create. This is why Turi Create is preferred by more experienced data scientists as it offers a level of customizability simply not available in Create ML.

What is Style Transfer?

Now that you have a fair understanding of Turi Create, let’s look at what Style Transfer is. Style transfer is the technique of recomposing images in the style of other images. What do I mean by this? Take a look at the image below created by using Prisma:

As you can see, the image of the breakfast plate above is transformed into the style of a comic book. Style Transfer began when Gatys et al. published a paper on how it was possible to use convolutional neural networks to transfer artistic style from one image to another.

Convolutional Neural Networks (CNNs) are a type of neural network in machine learning that are commonly used in areas such as image recognition and classification. CNNs have been successful in computer vision related problems like identifying faces, objects and more. This is fairly complex ideas so I wouldn’t worry too much about it.

Building our own Style Transfer Demo

Now that you have an understanding of the tools and concepts that we’ll examine in the tutorial, it is finally time to get started! We’ll be building our own Style Transfer model using Turi Create and import it to a sample iOS project to see how it’ll work!

First, download the starter project here. In this tutorial, we’ll be using Python 2, Jupyter Notebook, and Xcode 9.

Training the Style Transfer Model

Turi Create is a Python package, but it is not built into macOS. Therefore, let me go over how to install this really quick. Your macOS should have Python installed. In case you don’t have Python or pip installed on your device, you can learn the installation procedures over here.

Installing Turi Create and Jupyter

Open Terminal and type the following command:

pip install turicreate==5.0b2

Wait for a minute or two for the Python package to get installed. In the meantime, download Jupyter Notebook. Jupyter Notebook is a compiler for many languages used by developers because of its rich and interactive output visualization. Since Turi Create only supports Python 2, enter the following commands in Terminal to install Jupyter Notebook for Python 2.

python -m pip install --upgrade pip
python -m pip install jupyter

Once you have all the packages installed, it’s time to start creating our algorithm!

Coding with Turi Create

The style transfer model we’ll be creating is from Vincent van Gogh’s Starry Night. Basically, we’ll create a model which can transform any image into a replica of Starry Night.

First, download the training data and unzip it. One folder should be named content and the other should be named style. If you open content, you’ll see roughly 70 images with different subjects. This folder contains varied images so our algorithm knows what type of images to apply the style transfer to. Since we want the transformation on all images, we will have a variety of images.

Style, on the other hand, simply contains only one image: StarryNight.jpg. This folder contains the image we want the artistic style to transfer from.

Now, let’s start our coding session by opening Jupyter Notebook. Enter the following into Terminal.

jupyter notebook

This will open up Safari with the page like below.

Select the New button and click on Python 2!

Once you click on that button, a new screen should open. This is where we will create our model. Click on the first cell and begin by importing the Turi Create package:

import turicreate as tc

Press SHIFT+Enter to run the code in that cell. Wait until the package is imported. Next, let’s create a reference to the folders which contain our images. Please make sure you change the parameters to your own folder paths.

style = tc.load_images('/Path/To/Folder/style')
content = tc.load_images('/Path/To/Folder/content')

Run the code in the text field and you should receive an output like this:

Don’t worry too much about the warnings. Next, we’ll type in the command to create the style transfer model. It is highly advised that you run the following code on a Mac with a very powerful GPU! This would mean most of the latest MacBook Pros as well as the iMacs. If you choose to run the code on a MacBook Air, for example, the computations would run on the CPU and could take days!

model = tc.style_transfer.create(style, content)

Run the code. This could take a very long time to finish based on the device you are running this on. On my MacBook Air, it took 3.5 days since the computations were running on the CPU! If you don’t have enough time, no worries. You can download the final Core ML model here. However, you can always let the whole function run to get a feel for what it is like!

The table you see contains three columns: Iteration, Loss, and Elapsed Time. In Machine Learning, there will be a function which runs back and forth several times. When the function runs forward, this is known as cost. When it goes back, it is known as loss. Each time the function runs, the goal is to tweak the parameters to reduce the loss. So each time the parameters are altered, this adds one more iteration. The goal is to have a small number for the loss. As the training progresses, you can see the loss slowly reduces. Elapsed time refers to how long it has been.

When the model has finished training, all that’s left is saving it! This can be achieved with a simple line of code!

model.export_coreml("StarryStyle.mlmodel")

That’s all! Head over to your Library to view the final model!

A Quick Look at the Xcode Project

Now that we have our model, all that’s left is importing it to our Xcode project. Open Xcode 9 and take a look at the project.

Build and run the project to make sure you can compile the project. The app is not working right now. When we press the Van Gogh! button, nothing happens! It’s up to us to write the code. Let’s get started!

Implementing Machine Learning

The first step is to drag and drop the model file (i.e. StarryStyle.mlmodel) into the project. Make sure that Copy Items If Needed is checked and the project target is selected.

Next, we have to add the code to process the machine learning in ViewController.swift. Most of the code will be written in our transformImage() function. Let’s begin by importing the Core ML package and calling the model.

import CoreML
...
@IBAction func transformImage(_ sender: Any) {
    // Style Transfer Here
    let model = StarryStyle()
}

This line simply assigns our Core ML model to a constant called model.

Converting the Image

Next, we have to convert an image which a user chooses into some readable data. If you look into the StarryStyle.mlmodel file again, you should find that it takes in an image of size 256×256. Therefore, we have to perform the conversion. Right below our transformImage() function, add a new function.

func pixelBuffer(from image: UIImage) -> CVPixelBuffer? {
    // 1
    UIGraphicsBeginImageContextWithOptions(CGSize(width: 256, height: 256), true, 2.0)
    image.draw(in: CGRect(x: 0, y: 0, width: 256, height: 256))
    let newImage = UIGraphicsGetImageFromCurrentImageContext()!
    UIGraphicsEndImageContext()

    // 2
    let attrs = [kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue, kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue] as CFDictionary
    var pixelBuffer : CVPixelBuffer?
    let status = CVPixelBufferCreate(kCFAllocatorDefault, 256, 256, kCVPixelFormatType_32ARGB, attrs, &pixelBuffer)
    guard (status == kCVReturnSuccess) else {
        return nil
    }
       
    // 3   
    CVPixelBufferLockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
    let pixelData = CVPixelBufferGetBaseAddress(pixelBuffer!)
       
    // 4     
    let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
    let context = CGContext(data: pixelData, width: 256, height: 256, bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer!), space: rgbColorSpace, bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue)
       
    // 5
    context?.translateBy(x: 0, y: 256)
    context?.scaleBy(x: 1.0, y: -1.0)
    
    // 6 
    UIGraphicsPushContext(context!)
    image.draw(in: CGRect(x: 0, y: 0, width: 256, height: 256))
    UIGraphicsPopContext()
    CVPixelBufferUnlockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
        
    return pixelBuffer
}

This is a helper function, similar to the same function used in the earlier Core ML tutorial. In case you don’t remember, don’t worry. Let me go step by step to explain what this function does.

1. Since our model only accepts images with dimensions of 256 x 256, we convert the image into a square. Then, we assign the square image to another constant newImage.
2. Now, we convert newImage into a CVPixelBuffer. If you’re not familiar with CVPixelBuffer, it’s basically an image buffer which holds the pixels in the main memory. You can find out more about CVPixelBuffers here.
3. We then take all the pixels present in the image and convert them into a device-dependent RGB color space. Then, by creating all this data into a CGContext, we can easily call it whenever we need to render (or change) some of its underlying properties. This is what we do in the next two lines of code by translating and scaling the image.
4. Finally, we make the graphics context into the current context, render the image, and remove the context from the top stack. With all these changes made, we return our pixel buffer.

This is really some advanced Core Image code, which is out of the scope of this tutorial. Don’t worry if you didn’t understand most of it. The gist is that this function takes an image and extracts its data by turning it into a pixel buffer which can be read easily by Core ML.

Applying Style Transfer to the Image

Now that we have our Core ML helper function in place, let’s go back to transformImage() and implement the code. Below the line where we declare our model constant, insert the following code:

let styleArray = try? MLMultiArray(shape: [1] as [NSNumber], dataType: .double)
styleArray?[0] = 1.0

Turi Create allows you to package more than one “style” into a model. For this project, we only have one style: Starry Night. However, if you wanted to add more styles, you could add more pictures to the style folder. We declare styleArray as an MLMultiArray. This is a type of array used by Core ML for either an input or an output of a model. Since we have one style, we only have one shape and one data element. This is why we set the number of data elements to 1 for our styleArray.

Finally, all that’s left is making a prediction using our model and setting it to the imageView.

if let image = pixelBuffer(from: imageView.image!) {
    do {
        let predictionOutput = try model.prediction(image: image, index: styleArray!)
                
        let ciImage = CIImage(cvPixelBuffer: predictionOutput.stylizedImage)
        let tempContext = CIContext(options: nil)
        let tempImage = tempContext.createCGImage(ciImage, from: CGRect(x: 0, y: 0, width: CVPixelBufferGetWidth(predictionOutput.stylizedImage), height: CVPixelBufferGetHeight(predictionOutput.stylizedImage)))
        imageView.image = UIImage(cgImage: tempImage!)
    } catch let error as NSError {
        print("CoreML Model Error: \(error)")
    }
}

This function first checks if there is an image in imageView. In the code block, it defines a predictionOutput to save the output of the model’s prediction. We call the model’s prediction method with the user’s image and the style array. The predicted result is a pixel buffer. However, we can’t assign set a pixel buffer to a UIImageView so we come up with a creative way to do so.

First, we set the pixel buffer, predictionOutput.stylizedImage, to an image of type CIImage. Then, we create a variable tempContext which is an instance of CIContext. We call upon a built-in function of the context (i.e. createCGImage) which generates a CGImage from ciImage. Finally, we can set imageView to tempImage. That’s all! If there is an error, we gracefully handle it by printing the error.

Build and run your project. Choose an image from your photo library and test how the app works!

You may notice that the model may not look too close to Starry Night and this can be due to a variety of reasons. Maybe we need more training data? Or perhaps we need to train the model for a higher (or lower) number of iterations? I highly encourage you to go back and play around with the numbers until you get a satisfactory result!

Conclusion

This sums up the tutorial! I have given you an introduction to Turi Create and created your own Style Transfer model, a feat that would have been impossible for a single person to create just 5 years ago. You also learned how to import this Core ML model into an iOS app and use it for creative purposes!

However, Style Transfer is just the beginning. As I mentioned earlier, Turi Create can be used for a variety of applications. Here are some great resources on where to go next:

• Apple’s Gitbook on Turi Create Applications
• A Guide to Turi Create – WWDC 2018
• Turi Create Repository

For the full project, please download it from GitHub. If you have any comments or feedback, please leave me comment and share your thought below.

At WWDC 2018, Apple released ARKit 2.0 with a slew of brand new APIs and features for Augmented Reality Development. One of these features was an addition to their Quick Look APIs. If you’re not familiar with what Quick Look is, it’s basically a framework that allows users to preview a whole range of file formats such as PDFs, images, and more! For example, the Mail application in iOS uses Quick Look to preview attachments.

An advantage about using Quick Look in your apps is that all you need to do is state what file you would like to be quick looked. The framework handles displaying the UI and UX which makes it easy to integrate. Before going on, I would suggest to skim over this tutorial on using the Quick Look framework to preview documents.

This year, for iOS 12, Apple has introduced Quick Look for Augmented Reality objects. This means that you can share .usdz (more on that later) files in Mail, Messages, or any application that supports this type of Quick Look. The recipient can open it up and view the object without having to download an additional app.

In the above image, you can see a teapot being previewed in AR without the help of a separate app. However, AR Quick Look isn’t limited to simply apps. You can integrate AR Quick Look into websites as well! In this tutorial, I’ll walk you through integrating AR Quick Look in an iOS application, as well as, building a very basic HTML website using GitHub Pages to see how we can add AR Quick Look to websites! Just check out Apple’s AR Gallery on any device running iOS 12!

What is USDZ?

Before we start coding, it’s important to understand what USDZ is. Just like there are many file formats, USDZ is one of them. It stands for Universal Scene Description Zip. If you’ve worked with 3D models before, you’re proabably familiar with 3D models like .OBJ, .DAE, or .sketchup. USDZ is a file created from a collaboration with Pixar and Apple.

At its core, a USDZ file is nothing more than a .zip archive, that it packages the model and its textures into a single file. This is why USDZ files are used for Quick Look and not any other 3D model format.

Now you’re probably wondering, “How do I create a USDZ file?” Well, the way it works is that you create your own 3D model with your favorite 3D modelling software (AutoCAD, Blender, Maya, etc.) and then use Xcode Command Line Tools to convert this file to a .usdz file format.

Let’s try converting our own model into the USDZ file format!

Converting a 3D model to a USDZ file format

Converting a model to USDZ is quite simple and requires only one line of code! We’ll be converting a 3D Object model of an egg that I created. You can download the model here.

When you download the model, you’ll see that it’s simply an egg. Now, let’s try converting the model into a USDZ file format. Open Terminal and type the following line:

xcrun usdz_converter /Users/You/PATH/TO/egg.obj /Users/You/CHOOSE/A/PATH/TO/SAVE/egg.usdz

That’s all! Here’s what my terminal looks like:

Press enter. Within a few seconds, you will see the .usdz file saved to the path you chose where you want to save it. Press the spacebar to Quick Look the file.

Don’t worry if it’s black. I’m not sure if this is a bug on Apple’s part or if they meant it to be like this. Either way, when you preview it, all the colors will be restored.

And that’s all! Now that we have our own USDZ file, let’s begin with integrating it into our project.

Adding AR Quick Look in your apps

Let’s begin by downloading the starter project here. Take a look around. You’ll see that there is already a collection view in place.

Run the app. You’ll see a list of a bunch of models but when you click on them, nothing happens. It’s up to us to make sure that users can quick look the model!

First, let’s add our Egg model to the Models folder. Drag egg.usdz to the models folder. Make sure that when you drop it into the folder, you check the target box as shown below.

Next, head over to ViewController.swift and add egg to the models array. This way when we run our app, the model will show up in the list. Just to be sure, run the app again.

It works! Now all that’s left is adding the code to Quick Look these models. First, begin by importing the QuickLook package. When we create a UICollectionView, we also add the Data Source and Delegate protocols to give us access to the methods needed to add data to our collection view. Similarly, we do the same for Quick Look. Modify your code to look like below.

import UIKit
import Foundation
import QuickLook

class ViewController: UIViewController, UICollectionViewDelegate, UICollectionViewDataSource, QLPreviewControllerDelegate, QLPreviewControllerDataSource

There are two methods we need to add in order to configure to the protocols we added: numberOfPreviewItems() and previewController(previewItemAt). This should look familiar to you if you’ve worked with UITableViews or UICollectionViews. Add the following code towards the bottom of the class below collectionView(didSelectItemAt).

func numberOfPreviewItems(in controller: QLPreviewController) -> Int {
    return 1
}
    
func previewController(_ controller: QLPreviewController, previewItemAt index: Int) -> QLPreviewItem {
    let url = Bundle.main.url(forResource: models[thumbnailIndex], withExtension: "usdz")!
    return url as QLPreviewItem
}

1. In the first function, we are asked how many items are allowed to preview at a time. Since we want to preview one 3D model at a time, we return 1 in the code.
2. The second function asks what file it should preview when an item is clicked on a particular index. We define a constant called url which is the path of our .usdz files. Then, we return the file as a QLPreviewItem.

If you run the code now, nothing would happen. Why? It’s because we never added the logic to present the QLPreviewController. Navigate to the collectionView(didSelectItemAt) method and modify it to look like the following:

func collectionView(_ collectionView: UICollectionView, didSelectItemAt indexPath: IndexPath) {
    thumbnailIndex = indexPath.item

    let previewController = QLPreviewController()
    previewController.dataSource = self
    previewController.delegate = self
    present(previewController, animated: true)
}

Just as I mentioned earlier, we set thumbnailIndex to the index the user clicks on. This helps the Quick Look Data Source methods know what model to use. If you are using Quick Look in your apps for any file type, you will always present it in a QLPreviewController. Whether it is a document, an images, or, in our case, a 3D model, the QuickLook framework requires you to present these files in a QLPreviewController. We set the previewController data source and delegates to self and then present it!

Here is what all the Quick Look code should look like:

Build and run your app. Make sure to try the app on a real device running iOS 12. Running the app on a simulator won’t present the Quick Look preview.

It works as expected! You now should know how to integrate AR Quick Look into your apps. But that’s not all since AR Quick Look also provides web support! In the next section, I’ll guide you through building a website using HTML and AR Quick Look.

Adding AR Quick Look to Websites in HTML

Now that we have an iOS app working, let’s build a similar feature on a website using HTML. If you’ve never worked with HTML before, don’t worry! I’ll guide you through building a very simple website. Let’s get started!

To begin, open any text editor on your Mac. This can be TextEdit or any other similar application. I will be using TextMate. Type the following:

<!DOCTYPE html>
<html>

</html>

This is the way you begin all HTML websites. The <!DOCTYPE html> is an instruction to the web browser about what HTML version the page is written in. We are using HTML 5.

The angled brackets are known as tags. Similar to how we declare all our code in a class in Swift, all HTML code must be declared in between the <html> and </html> tags. The <html> tag represents the start of the HTML code where as the </html> tag signifies the end.

Everytime you visit a website, you’ll see the title of that website in the tab.

Let’s see how to write this in HTML! In between the HTML tags, type the following code:

<head>
    <title>AR Library</title>
</head>

The <head> tag is a place where all the metadata about a website is stored. Some examples of metadata include built-in links, scripts, favicons, or in our case, the title.

Since we’re defining the title of our website, we put the title in between the <title> and </title> tags.

One thing you’ll notice about HTML is that strings don’t require you to put quotations around them. This is one of my favorite aspects of HTML.

Your text file should look like this:

Now, we have to define the body of our website. This means all the text, button, and images you’ll see in a website. Like before, we declare these in the <body> tag. Right under the </head> line, we’ll add the following code:

<body>
    <h1>Welcome to the AR Library</h1>
    <p>Welcome to the AR Library website. I created this website in order to view AR objects from the web on any device running iOS 12. Conincidentally, this is the first time I made a website with HTML! It's a lot of fun!</p>
</body>

Inside our <body> tag, you should see two new tags: <h1> and <p>. H1 stands for Header 1. This is usually for a title of a section. P stands for paragraph. This tag is used when you want to write some long body of text. Remember, you can rename the title and paragraph to whatever you want!

Save your file. Make sure that when you do, you use a .html ending.

Click on the file you saved and it should open in Safari (or your default browser)!

Congratulations! You just built your first website in HTML!

You’re probably wondering if you can change the font and font size. This is possible through CSS. Currently that is beyond the scope of the tutorial but you can find a great article here.

Adding AR Buttons

Now that we have some text for our website, let’s add the buttons for users to launch the AR Quick Look view in the website. Since we are making a button, this will still be inside the <body> tag of our code. Below the <p> tag, type the following.

<a href="egg.usdz" rel="ar">
    <img src="egg.png" width=200>
</a>

This is the <a> tag which defines a hyperlink. There are several customizeable attributes under the <a> tag which we define above.

1. The first attribute is href. This is basically a path to the document we want to navigate to when our button is clicked on. The “document” is our 3D model, so I put the name of a .usdz file there.
2. The second is rel. This specifies the relationship between the current page we’re on and the page we link to. I set it to ar because the relationship of egg.usdz is an AR model.
3. Now we have our button defined but we didn’t define what our button should look like. By using the <img src> tag, I’m defining the image our button should have. This way when our users click on the image, they’ll be directed to the AR Quick Look view. I also set the width of my image so that it’s not too big. The image I’m using is the one we already have in our Xcode project.

That’s it! You can add the other buttons in a similar manner.

When referencing your image source and USDZ files, make sure that they are in the same folder as your HTML file.

Open the file on your web browser. Take a look- your first HTML website hosting a powerful AR feature!

However, when you click on an image, it only directs you to the actual folder on your device. Plus, there’s no way to view it on an iPhone or iPad. This is where GitHub Pages comes in!

Uploading to GitHub Pages

GitHub Pages is a great way to host static websites. Many people use GitHub Pages as a way to showcase either a résumé or an About page for a project or organization.

One of the advantages about GitHub Pages is that it can be edited from a repository on your account. Through this, it’s a great way to store files (such as images and AR models) and reference them in your website! Let’s explore how this can be done! If you haven’t one already, create a GitHub account here.

Once you have an account, go to the home page and click on the Plus button in the upper right corner. Click on New repository.

The way GitHub Pages works is that you’re given only one domain: username.github.io. Any pages you create are subdomains under the URL. Therefore, name your repository username.github.io. In the image below, you can see that I named my repository aidev1065.github.io, since aidev1065 is my username. You can leave the rest of the settings and click on Create Repository.

When you’re presented with the repository page, navigate to the Settings tab and scroll down until you come to the GitHub Pages section.

Click on Choose a theme under “Theme Chooser”. This creates a theme for our page. There are a variety of themes. You can choose the one which suits you but I’m going to go with Cayman!

After you click on Select theme, you’ll be presented with a Markdown file. This is used to present information on our website. If you’re not familiar with Markdown syntax, don’t worry. It’s not that big of a deal. Just delete everything in the index.md file and type the following:

# AR Library
This is a website for an AR Library! You can view it [here](Website.html)!

What’s important to understand here is that in the parenthesis, put the name of the HTML file you created earlier!

Scroll to the bottom and click on Commit changes. Now you have your Markdown file ready! If you go to any web browser and type in “username.github.io”, you’ll be directed to your own GitHub Page!

However, when we click on “here”, we get an invalid page error. This is because we have not uploaded the HTML file, USDZ files, and images! Let’s do that now!

Head on back to the repository page and click on the button Upload files.

Now all that’s left is uploading the HTML files, the USDZ models, and the images! There should be 19 files in total: 1 HTML file, 9 USDZ models, and 9 images.

Scroll to the bottom and click on Commit changes. This will take a few minutes. When everything is done, go back to your GitHub Page: “username.github.io”. Now, when you click on “here”, you’ll see the HTML website you created earlier.

Also, when you open the website on a device running iOS 12, you’ll see the ARKit logo in the top right of the image. This means that you can Quick Look the model!

When you click any of the images, you’re presented with the same viewer as that of the iOS app!

Conclusion

Congratulations for making it this far! I hope you have enjoyed and learned something valuable from my tutorial. You should now understand how to convert 3D models to USDZ files, integrate them in your apps using QLPreviewController, use HTML to build a basic website, and use GitHub Pages to host your files. Feel free to share this tutorial on your social networks so that your friends can learn too!

Here are some resources related to the tutorial that can help you expand this project:

• Integrating Apps and Content with AR Quick Look – WWDC 2018
• Quick Look Framework Documentation
• HTML Tutorial
• CSS Tutorial
• Official GitHub Pages Page

For reference, you can download the complete Xcode project on GitHub here and visit the repository I created in the tutorial here.

Core ML is Apple’s Machine Learning framework. Released just a year ago, Core ML offers developers a way to integrate powerful and smart machine learning capabilities into their apps with just a few lines of code! This year, at WWDC 2018, Apple released Core ML 2.0- the next version of Core ML all centered around streamlining the process by optimizing the size of the model, improving the performance, and giving developers the ability to customize their own Core ML models.

In this tutorial, I’ll catch you up on all of the new features introduced in Core ML 2.0 and how you can apply it to your Machine Learning app! In case you are new to Core ML, I would recommend getting familiar with Core ML with this tutorial. In case you are familiar with it, let’s get started!

A Quick Recap

There are lots of great apps in the App Store with the ability to perform powerful tasks. For example, you could find an app which understands text. Or maybe an app which knows what workout you’re doing based on the motion of your device. Even further, there are apps which apply filters to images based on a previous image. These apps have one thing in common: they are all examples of machine learning and all of them can be created with a Core ML model.

Source: Apple

Core ML makes it simple for developers to integrate machine learning models into their apps. You can create an app which understands context in a conversation or can recognize different audio. Furthermore, Apple has made it possible for developers to take the extra step with realtime image analysis and natural language understanding through 2 of their frameworks: Vision and Natural Language.

With the VNCoreMLRequest API and the NLModel API, you can heavily increase your app’s ML capabilities since Vision and Natural Language are built upon Core ML!

• Vision Tutorial
• Natural Language Tutorial

This year, Apple was focused on 3 main points for helping Core ML developers.

1. The model size
2. The performance of a model
3. Customizing a model

Let’s explore these three points!

Model Size

One huge advantage of Core ML is that everything is done on-device. This way, a user’s privacy is always safe and the computation can be calculated from anywhere. However, as more accurate machine learning models are used, they can have a larger size. Importing these models into your apps can take up a large amount of space on a user’s device.

Apple decided to give developers the tools to quantize their Core ML models. Quantizing a model refers to the techniques used to store and calculate numbers in a more compact form. At the core roots of any machine learning model, it’s just a machine trying to compute numbers. If we were to reduce the numbers or store them in a form that would take less space, we can drastically reduce the size of a model. This can lead to a reduced runtime memory usage and faster calculations!

There are 3 main parts to a machine learning model:

• The number of models
• The number of weights
• The size of the weights

When we quantize a model, we are reducing the size of the weight! In iOS 11, Core ML models were stored in 32-bit models. With iOS 12 and Core ML 2, Apple has given us the ability to store the model in 16-bit and even 8-bit models! This is what we’ll be looking at in this tutorial!

In case you aren’t familiar with what weights are, here’s a really good analogy. Say that you’re going from your house to the supermarket. The first time, you may take a certain path. The second time, you’ll try to find a shorter path to the supermarket, since you already know your way to the market. And the third time, you’ll take an even shorter route because you have the knowledge of the previous 2 paths. Each time you go to the market, you’ll keep taking a shorter path as you learn over time! This knowledge of knowing which route to take is known as the weights. Hence, the most accurate path, is the one with the most weights!

Let’s put this into practice. Time for some code!

Weight Quantization

As an example, let’s use a popular machine learning model called Inception v3 for demo. You can download the model in the Core ML format here.

Opening this model, you can see that it takes up quite a bit of space at 94.7 MB.

We will use the Python package coremltools to quantize this model. Let’s see how!

In case you don’t have python or pip installed on your device, you can learn the installation procedures over here.

First, you need to make sure to install the beta version of coremltools. Open terminal and type in the following:

pip install coremltools==2.0b1

You should see an output like this.

This command successfully installs the beta version of Core ML Tools beta 1. The next few steps require some Python. No worries however, it’s really simple and doesn’t require too much code! Opening up a Python editor of your choice or follow along in the Terminal. First, let’s import the coremltools package. Type python in the Terminal and then type the following once the editor shows up:

import coremltools
from coremltools.models.neural_network.quantization_utils import *

This imports the Core ML tools package, as well as, all the quantization utilities. Next, let’s define a variable model and set its URL to the Inceptionv3.mlmodel just downloaded.

model = coremltools.models.MLModel('/PATH/TO/Inceptionv3.mlmodel')

Before we quantize the model (it takes only 2 lines!), let me give you some background information of neural network.

A neural network is composed of different layers. These layers are nothing but math function which have many parameters. These parameters are known as weights.

Source: Towards Data Science

When we quantize weights, we take the minimum value of a weight and the maximum value of a weight and map them. There are many ways to map them but the most commonly used ones are linear and lookup. Linear Quantization is when you map the weights evenly and reduce them. In a Lookup Table Quantization, the model constructs a table and groups the weights around based on similarity and reduces them.

If this sounds complicated, don’t worry. All we need to do is choose the number of bits we want our model to be represented by and the algorithm to choose. First, let’s see what happens if we choose linearly quantize a model.

lin_quant_model = quantize_weights(model, 16, "linear")

In the above code, we quantize the weights of the Inceptionv3 model to 16 bits and use linear quantization. Running the code should give you a long list of every layer the program is quantizing.

Let’s save the model and see how it compares to the original model. Choose a path where to save it and type the following.

lin_quant_model.save('Path/To/Save/QuantizedInceptionv3.mlmodel')

Now open both models and compare the sizes.

When we represent the Inceptionv3 model in a 16-bit format, it takes up less space!

However, it is important to remember what weight quantization really is. Earlier, in my analogy, I said that more weights yields more accuracy. When we quantize a model, we are also reducing the accuracy of the model along with the size. Quantization is an accuracy tradeoff. Quantized models are approximations of the size of the weight, so it is always important to run your quantized models and see how they perform.

Ideally, we want to quantize our models while retaining the highest possible accuracy. This can be done by finding the right quantization algorithm. In the previous example, we used Linear Quantization. Now let’s try to use Lookup Quantization now and see what happens. Just like before, type the following into Terminal:

lut_quant_model = quantize_weights(model, 16, "kmeans")

When the model is done quantizing, we need to compare both our lin_quant_model and our lut_quant_model to the original model by running it through some sample data. This way, we can find out which quantized model was most similar to the original model. Download a folder of sample images here. Type the following line and we can see which model performed better!

compare_models(model, lin_quant_model, '/Users/SaiKambampati/Desktop/SampleImages')

This may take a while but after both models are done processing, you will receive an output which looks like this:

We are interested in the Top 1 Agreement. It shows 100% which means that it matches 100% with our model! This is great for us as we now have a quantized model which takes less space and has approximately the same accuracy as our original model! We can import this into a project now if we want but let’s compare the Lookup Table Quantized model as well!

compare_models(model, lut_quant_model, '/Users/SaiKambampati/Desktop/SampleImages')

We receive an output of 100% as well so both models are compatible! I encourage you to play around with quantizing different models. In the above example, we quantized the Inceptionv3 model down to a 16-bit model. See if you can continue to quantize the model to an 8-bit representation and even a 4-bit representation and compare it with the sample data! How did it perform?

The above image depicts what happened when I quantized the Inceptionv3 model using a linear algorithm to a 1-bit representation! As you can see, the model size drastically reduced but so did the accuracy! In fact, it is completely inaccurate with a 0% accuracy. Play around with quantization to try and find the happy medium. Always remember to test your quantized models to make sure they perform accurately!

Performance

The next point Apple focused on with Core ML 2 was performance. Since we are running the ML computations on-device, we want it to be fast and accurate. This can be pretty complicated. Luckily, Apple has provided us with a way to improve the performance of our CoreML models. Let’s walk through an example.

Style Transfer is a machine learning application which basically transforms a certain image into the style of another. If you’ve the Prisma app before, that’s a sample use of Style Transfer.

If we were to look into the neural network of style transfer, this is what we would notice. There are a certain set of inputs for this algorithm. Each layer in the neural network adds a certain transformation to the original image. This means that the model has to take in every input and map it out to an output and make a prediction from that. The prediction then helps in the creation of the weights. What would this look like in code?

// Loop over inputs
for i in 0..< modelInputs.count {
    modelOutputs[i] = model.prediction(from: modelInputs[i], options: options) 
}

In the above code, you see that for each input, we are asking the model to generate a prediction and yield an output based on some options. However, iterating over every input can take a long time.

To combat this, Apple has introduced a brand new Batch API! Unlike a for loop, a batch in machine learning is when all the inputs are fed to the model and an accurate prediction is its result! This can take much less time and more importantly, much less code!

Here is the above for-loop code written with the new Batch Predict API!

modelOutputs = model.prediction(from: modelInputs, options: options)

That’s all! Just one line of code is all it takes! You’re probably wondering, “Wait! I never did this before? This sounds complicated. Where do I even use this?” That bring me to my last point which is Customization.

Customization

When you open the hood of a neural network, you see that they are comprised of many layers. However, there may be some scenarios when you are trying to convert a neural network from Tensorflow to Core ML. Or maybe a pipeline from Keras to Core ML. However, there may be the occasional example where Core ML simply does not have the tools to convert the model correctly! What do I mean by this? Let’s take another example.

Image recognition models are built using a Convolutional Neural Network (CNN). CNNs consists of series of layers which are highly optimized. When you convert a neural network from one format to Core ML, you are transforming each layer. However, there may be some rare scenario where Core ML simply does not provide the tools to convert a layer. In the past, you could not do anything about it but with iOS 12, the Apple engineers have introduced the MLCustonLayer protocol which allows developers to create their own layers in Swift. With MLCustomLayer, you can define the behavior of your own neural network layers in Core ML models. However, it’s worth noting that custom layers only work for neural network models.

Now if this sounds very complicated, don’t worry. It usually takes a skilled data scientists or an machine learning engineer to understand all the intricacies of a neural network and have the talent of writing their own model. This is beyond the scope of the tutorial so we won’t delve into this.

Conclusion

That sums up all the new changes in Core ML 2.0. Core ML 2.0 aims to make models smaller, faster, and more customizable. We saw how to reduce the size of our Core ML models through weight quantization, improve the performance of our model through the new Batch API, and examples where we might need to write custom layers for our model. As developers, I predict (see what I did there) that you’ll be using weight quantization more than the other two techniques (Batch API and Custom Layers).

If you are interested in exploring Core ML 2.0 even more, here are some excellent resources to look over!

• What’s New in Core ML, Part 1 – WWDC 2018
• What’s New in Core ML, Part 2 – WWDC 2018
• Vision with Core ML – WWDC 2018
• Introducing Natural Language Framework – WWDC 2018
• Core ML Documentation
• Apple’s Machine Learning Page

Just as Apple does a lot for its developer community, another company which goes to great lengths to create amazing tools and services for its developers is Google. In recent years, Google has released and improved its services such as Google Cloud, Firebase, TensorFlow, etc. to give more power to both iOS and Android developers.

This year at Google I/O 2018, Google released a brand new toolkit called ML Kit for its developers. Google has been at the front of the race towards Artificial Intelligence and by giving developers access to its ML Kit models, Google has put a lot of power into its developers.

With ML Kit, you can perform a variety of machine learning tasks with very little code. One core difference between CoreML and ML Kit is that in CoreML, you are required to add your own models but in ML Kit, you can either rely on the models Google provides for you or you can run your own. For this tutorial, we will be relying on the models Google uses since adding your own ML models requires TensorFlow and an adequate understanding of Python.

Another difference is that if your models are large, you have the ability to put your ML model in Firebase and have your app make calls to the server. In CoreML, you can only run machine learning on-device. Here’s a list of everything you can do with ML Kit:

• Barcode Scanning
• Face Detection
• Image Labelling
• Text Recognition
• Landmark Recognition
• Smart Reply (not yet released, but coming soon)

In this tutorial, I’ll show you how to create a new project in Firebase, use Cocoapods to download the required packages, and integrate ML Kit into our app! Let’s get started!

Creating a Firebase Project

The first step is to go to the Firebase Console. Here you will be prompted to login to your Google Account. After doing so, you should be greeted with a page that looks like this.

Click on Add Project and name your project. For this scenario, let’s name this project ML Kit Introduction. Leave the Project ID as is and change the Country/Region as you see fit. Then, click on the Create Project button. This should take about a minute.

When everything is all done, your page should look like this!

This is your Project Overview page and you’ll be able to manipulate a wide variety of Firebase controls from this console. Congratulations! You just created your first Firebase project! Leave this page as is and we’ll shift gears for a second and look at the iOS project. Download the starter project here.

A Quick Glance At The Starter Project

Open the starter project. You’ll see that most of the UI has been designed for you already. Build and run the app. You can see a UITableView with the different ML Kit options which lead to different pages.

If you click on the Choose Image button, a UIImagePickerView pops up and choosing an image will change the empty placeholder. However, nothing happens. It’s up to us to integrate ML Kit and perform its machine learning tasks on the image.

Linking Firebase to the App

Go back to the Firebase console of your project. Click on the button which says “Add Firebase to your iOS App”.

You should receive a popup now with instructions on how to link Firebase. The first thing to do is to link enter your iOS Bundle ID. That can be found in the Project Overview of Xcode in the General tab.

Enter that into the field and click the button titled Register App. You don’t need to enter anything into the optional text fields as this app is not going to go on the App Store. You’ll be guided to Step 2 where you are asked to download a GoogleService-Info.plist. This is an important file that you will add to your project. Click on the Download button to download the file.

Drag the file into the sidebar as shown on the Firebase website. Make sure that the Copy items if needed checkbox is checked. IF you’ve added everything, click on the Next button and proceed to Step 3.

Installing Firebase Libraries Using Cocoapods

This next step will introduce the idea of Cocoapods. Cocoapods are basically a way for you to import packages into your project in an easy manner. However, there can be a lot of disastrous consequences if any slight error is made. First, close all windows on Xcode and quit the application.

Open Terminal on your Mac and enter the following command:

cd <Path to your Xcode Project>

You are now in that directory.

To create a pod, it’s quite simple. Enter the command:

pod init

Wait for a second and then your Terminal should look like this. Just a simple line was added.

Now, let’s add all the packages we need to our Podfile. Enter the command in terminal and wait for Xcode to open up:

open -a Xcode podfile

Underneath where it says # Pods for ML Kit Starter Project, type the following lines of code:

pod 'Firebase/Core'
pod 'Firebase/MLVision'
pod 'Firebase/MLVisionTextModel'
pod 'Firebase/MLVisionFaceModel'
pod 'Firebase/MLVisionBarcodeModel'
pod 'Firebase/MLVision'
pod 'Firebase/MLVisionLabelModel'

Your Podfile should look like this.

Now, there’s only one thing remaining. Head back to Terminal and type:

pod install

This will take a couple of minutes so feel free to take a break. In the meantime, Xcode is downloading the packages which we will be using. When everything is done and you head back over to the folder where your Xcode project is, you’ll notice a new file: a .xcworkspace file.

This is where most developer mess up: YOU SHOULD NEVER AGAIN OPEN THE .XCODEPROJ FILE! If you do and edit content there, the two will not be in sync and you will have to create a new project all over again. From now on, you should always open the .xcworkspace file.

Go back to the Firebase webpage and you’ll notice that we have finished Step 3. Click on the next button and head over to Step 4.

We are now being asked to open our workspace and add a few lines of code to our AppDelegate.swift. Open the .xcworkspace (again, not the .xcodeproj. I cannot iterate over how important this is) and go to AppDelegate.swift.

Once we’re in AppDelegate.swift, all we need to do is add two lines of code.

import UIKit
import Firebase

@UIApplicationMain
class AppDelegate: UIResponder, UIApplicationDelegate {

    var window: UIWindow?

    func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplicationLaunchOptionsKey: Any]?) -> Bool {
        // Override point for customization after application launch.
        FirebaseApp.configure()
        return true
    }

All we are doing is importing the Firebase package and configuring it based on our GoogleService-Info.plist file we added earlier. You may get an error saying that it was not able to build the Firebase module but just press CMD+SHIFT+K to clean the project and then CMD+B to build it.

If the error persists, go to the Build Settings tab of the project and search for Bitcode. You’ll see an option called Enable Bitcode under Build Options. Set that to No and build again. You should be successful now!

Press the Next button on the Firebase console to go to Step 5. Now, all you need to do is run your app on a device and Step 5 will automatically be completed! You should then be redirected back to your Project Overview page.

Congratulations! You are done with the most challenging part of this tutorial! All that left is adding the ML Kit code in Swift. Now it would be a perfect time to take a break but from now on, it’s just smooth cruising in a familiar code we know!

Barcode Scanning

The first implementation will be with Barcode Scanning. This is really simple to add to your app. Head to BarcodeViewController. You’ll see the code which happens when the Choose Image button is clicked. To get access to all the ML Kit protocols, we have to import Firebase.

import UIKit
import Firebase

Next, we need to define some variables which will be used in our Barcode Scanning function.

let options = VisionBarcodeDetectorOptions(formats: .all)
lazy var vision = Vision.vision()

The options variable tells the BarcodeDetector what types of Barcodes to recognize. ML Kit can recognize most common formats of barcodes like Codabar, Code 39, Code 93, UPC-A, UPC-E, Aztec, PDF417, QR Code, etc. For our purpose, we’ll ask the detector to recognize all types of formats. The vision variable returns an instance of the Firebase Vision service. It is through this variable that we perform most of our computations through.

Next, we need to handle the recognition logic. We will implement all of this in the imagePickerController:didFinishPickingMediaWithInfo function. This function runs when we have picked an image. Currently, the function only sets the imageView to the image we choose. Add the following lines of code below the line imageView.image = pickedImage.

// 1
let barcodeDetector = vision.barcodeDetector(options: options)
let visionImage = VisionImage(image: pickedImage)

//2
barcodeDetector.detect(in: visionImage) { (barcodes, error) in
    //3
    guard error == nil, let barcodes = barcodes, !barcodes.isEmpty else {
        self.dismiss(animated: true, completion: nil)
        self.resultView.text = "No Barcode Detected"
        return
    }
    
    //4
    for barcode in barcodes {
        let rawValue = barcode.rawValue!
        let valueType = barcode.valueType
        
        //5
        switch valueType {
        case .URL:
            self.resultView.text = "URL: \(rawValue)"
        case .phone:
            self.resultView.text = "Phone number: \(rawValue)"
        default:
            self.resultView.text = rawValue
        }
    }
}

Here’s a quick rundown of everything that went down. It may seem like a lot but it’s quite simple. This will be the basic format for everything else we do for the rest of the tutorial.

1. The first thing we do is define 2 variables: barcodeDetector which is a barcode detecting object of the Firebase Vision service. We set it to detect all types of barcodes. Then we define an image called visionImage which is the same image as the one we picked.
2. We call the detect method of barcodeDetector and run this method on our visionImage. We define two objects: barcodes and error.
3. First, we handle the error. If there is an error or there are no barcodes recognized, we dismiss the Image Picker View Controller and set the resultView to say “No Barcode Detected”. Then we return the function so the rest of the function will not run.
4. If there is a barcode detected, we use a for-loop to run the same code on each barcode recognized. We define 2 constants: a rawValue and a valueType. The raw value of a barcode contains the data it holds. This can either be some text, a number, an image, etc. The value type of a barcode states what type of information it is: an email, a contact, a link, etc.
5. Now we could simply print the raw value but this wouldn’t provide a great user experience. Instead, we’ll provide custom messages based on the value type. We check what type it is and set the text of the resultView to something based on that. For example, in the case of a URL, we have the resultView text say “URL: ” and we show the URL.

Build and run the app. It should work amazingly fast! What’s cool is that since resultView is a UITextView, you can interact with it and select on any of the detected data such as numbers, links, and emails.

Face Detection

Next, we’ll be taking a look at Face Detection. Rather than draw boxes around faces, let’s take it a step further and see how we can report if a person is smiling, whether their eyes are open, etc.

Just like before, we need to import Firebase and define some constants.

import UIKit
import Firebase
...
let options = VisionFaceDetectorOptions()
lazy var vision = Vision.vision()

The only difference here is that we initialize the default FaceDetectorOptions. And, we configure these options in the viewDidLoad method.

override func viewDidLoad() {
    super.viewDidLoad()
    imagePicker.delegate = self
    
    // Do any additional setup after loading the view.
    options.modeType = .accurate
    options.landmarkType = .all
    options.classificationType = .all
    options.minFaceSize = CGFloat(0.1)
}

We define the specific options of our detector in viewDidLoad. First, we choose which mode to use. There are two modes: accurate and fast. Since this is just a demo app, we’ll use the accurate model but in a situation where speed is important, it may be wiser to set the mode to .fast.

Next, we ask the detector to find all landmarks and classifications. What’s the difference between the two? A landmark is a certain part of the face such as the right cheek, left cheek, base of the nose, eyebrow, and more! A classification is kind of like an event to detect. At the time of this writing, ML Kit Vision can detect only if the left/right eye is open and if the person is smiling. For our purpose, we’ll be dealing with classifications only (smiles and eyes opened).

The last option we configure is the minimum face size. When we say options.minFaceSize = CGFloat(0.1), we are asking for the smallest desired face size. The size is expressed as a proportion of the width of the head to the image width. A value of 0.1 is asking the detector to search for the smallest face which is roughly 10% of the width of the image being searched.

Next, we handle the logic inside the imagePickerController:didFinishPickingMediaWithInfo method. Right below the line imageView.image = pickedImage, type the following:

let faceDetector = vision.faceDetector(options: options)
let visionImage = VisionImage(image: pickedImage)
self.resultView.text = ""

This simply sets ML Kit Vision service to be a face detector with the options we defined earlier. We also define the visionImage to be the one we chose. Since we may be running this several times, we’ll want to clear the resultView.

Next, we call the faceDetector’s detect function to perform the detection.

//1
faceDetector.detect(in: visionImage) { (faces, error) in
    //2
    guard error == nil, let faces = faces, !faces.isEmpty else {
        self.dismiss(animated: true, completion: nil)
        self.resultView.text = "No Face Detected"
        return
    }
    //3
    self.resultView.text = self.resultView.text + "I see \(faces.count) face(s).\n\n"
    
    for face in faces {
        //4
        if face.hasLeftEyeOpenProbability {
            if face.leftEyeOpenProbability < 0.4 {
                self.resultView.text = self.resultView.text + "The left eye is not open!\n"
            } else {
                self.resultView.text = self.resultView.text + "The left eye is open!\n"
            }
        }
        
        if face.hasRightEyeOpenProbability {
            if face.rightEyeOpenProbability < 0.4 {
                self.resultView.text = self.resultView.text + "The right eye is not open!\n"
            } else {
                self.resultView.text = self.resultView.text + "The right eye is open!\n"
            }
        }
        
        //5
        if face.hasSmilingProbability {
            if face.smilingProbability < 0.3 {
                self.resultView.text = self.resultView.text + "This person is not smiling.\n\n"
            } else {
                self.resultView.text = self.resultView.text + "This person is smiling.\n\n"
            }
        }
    }
}

This should look very similar to the Barcode Detection function we wrote earlier. Here’s everything that happens:

1. We call the detect function on our visionImage looking for faces and errors.
2. In case if there is an error or there are no faces, we set the resultView text to “No Face Detected” and return the method.
3. If faces are detected, the first statement we print to the resultView is the number of faces we see. You’ll be seeing \n a lot within the strings of this tutorial. This signifies a new line.
4. Then we deep dive into the specifics. If a face has a probability for the left eye being opened, we check to see what the probability is. In this case, I have set it to say that the left eye is closed if the probability is less than 0.4. The same goes for the right eye. You can change it to whatever value you wish.
5. Similarly, I check the smile probability and if it is less than 0.3, the person is most likely not smiling, otherwise, the face is smiling.

Build and run your code, see how it performs! Feel free to tweak the values and play around!

Image Labelling

Next, we’ll check out labelling images. This is much easier than face detection. Actually, you have two options with Image Labelling and Text Recognition. You can either have all of the machine learning done on-device (which is what Apple would prefer since all the data belongs to the user, it can run offline, and no calls are made to Firebase) or you can use Google’s Cloud Vision. The advantage here is that the model will automatically be updated and a lot more accurate since it’s easier to have bigger, accurate model size in the cloud than on device. However, for our purposes, we’ll be continuing everything we have done so far and just implement the on-device version.

See if you can implement it on your own! It’s quite similar to the previous two scenarios. If not, that’s alright! Here’s what we should do!

import UIKit
import Firebase
...
lazy var vision = Vision.vision()

Unlike before, we’ll be using the default settings for the label detector so we only need to define one constant. Everything else, like before, will be in the imagePickerController:didFinishPickingMediaWithInfo method. Under imageView.image = pickedImage, insert the following lines of code:

//1
let labelDetector = vision.labelDetector()
let visionImage = VisionImage(image: pickedImage)
self.resultView.text = ""

//2
labelDetector.detect(in: visionImage) { (labels, error) in
    //3
    guard error == nil, let labels = labels, !labels.isEmpty else {
        self.resultView.text = "Could not label this image"
        self.dismiss(animated: true, completion: nil)
        return
    }
    
    //4
    for label in labels {
        self.resultView.text = self.resultView.text + "\(label.label) - \(label.confidence * 100.0)%\n"
    }
}

This should start to look familiar. Let me briefly walk you over the code:

1. We define labelDetector which is telling ML Kit’s Vision service to detect labels from images. We define visionImage to be the image we chose. We clear the resultView in case we use this function more than once.
2. We call the detect function on visionImage and look for labels and errors.
3. If there is an error or ML Kit wasn’t able to label an image, we return the function telling the user that we couldn’t label the image.
4. If everything works, we set the text of resultView to be the label of the image and how confident ML Kit is with labelling that image.

Simple, right? Build and run your code! How accurate (or crazy) were the labels?

Text Recognition

We’re almost done! Optical Character Recognition, or OCR, has become immensely popular within the last two years in terms of mobile apps. With ML Kit, it’s become so much easier to implement this in your apps. Let’s see how!

Similar to image labelling, text recognition can be done via Google Cloud and through calls to the model in the cloud. However, we’ll work with the on-device API for now.

import UIKit
import Firebase
...
lazy var vision = Vision.vision()
var textDetector: VisionTextDetector?

Same as before, we call the Vision service and define a textDetector. We can set the textDetector variable to vision‘s text detector in the viewDidLoad method.

override func viewDidLoad() {
    super.viewDidLoad()
    imagePicker.delegate = self
    textDetector = vision.textDetector()
}

Next, we just handle everything in imagePickerController:didFinishPickingMediaWithInfo. As usual, we’ll insert the following lines of code below the line imageView.image = pickedImage.

//1
let visionImage = VisionImage(image: pickedImage)
textDetector?.detect(in: visionImage, completion: { (features, error) in
    //2
    guard error == nil, let features = features, !features.isEmpty else {
        self.resultView.text = "Could not recognize any text"
        self.dismiss(animated: true, completion: nil)
        return
    }
    
    //3
    self.resultView.text = "Detected Text Has \(features.count) Blocks:\n\n"
    for block in features {
        //4
        self.resultView.text = self.resultView.text + "\(block.text)\n\n"
    }
})

1. We set visionImage to be the image we chose. We run the detect function on this image looking for features and errors.
2. If there is an error or no text, we tell the user “Could not recognize any text” and return the function.
3. The first piece of information we’ll give our users is how many blocks of text were detected.
4. Finally, we’ll set the text of resultView to the text of each block leaving a space in between with \n\n (2 new lines).

Test it out! Try putting it through different types of fonts and colors. My results have shown that it works flawlessly on printed text but has a really hard time with handwritten text.

Landmark Recognition

Landmark recognition can be implemented just like the other 4 categories. Unfortunately, ML Kit does not support Landmark Recognition on-device at the time of this writing. To perform landmark recognition, you will need to change the plan of your project to “Blaze” and activate the Google Cloud Vision API.

However, this is beyond the scope of this tutorial. If you do feel up to the challenge, you can implement the code using the documentation found here. The code is also included in the final project.

Conclusion

This was a really big tutorial, so feel free to scroll back up and review anything you may not have understood. If you have any questions, feel free to leave me a comment below.

With ML Kit, you saw how easy it is to implement smart machine learning features into your app. The scope of apps which can be created is large, so here’s some ideas for you to try out:

• An app which recognizes text and reads them back to users with visual disabilities
• Face Tracking for an app like Try Not to Smile by Roland Horvath
• Barcode Scanner
• Search for pictures based on labels

The possibilities are endless, it’s all based on what you can imagine and how you want to help your users. The final project can be downloaded here. You can learn more about the ML Kit API’s by checking out their documentation. Hope you learned something new in tthis tutorial and let me knowhow it went in the comments below!

For the full project, you can check it out on GitHub.

In case you weren’t aware, Apple’s Worldwide Developers Conference happened this week! It was a big event with a lot of improvements to both the software and the current frameworks Apple currently has. One of these frameworks is Create ML.

Last year, Apple introduced Core ML: a quick way for you to import pre-trained machine learning models into your app with as little code as possible! This year, with Create ML, Apple is giving us developers the ability to create our own machine learning models straight into Xcode Playgrounds! All we need is some data and we’re good to go! As of right now, Create ML allows text, images, and tables as data. However, since this constitutes for most ML applications, this should serve your purpose well! I’ll show you how to create a ML model with all 3 of these types of data.

[Image source: Apple]

Why Create ML

You’re probably wondering, why should I prefer Create ML? This is because of what it’s capable of. Create ML harnesses the machine learning infrastructure built into the software. When you download iOS 12 or macOS Mojave, you are also downloading some machine learning frameworks. That way, when you create your own ML model, it can take up less space since most of the data is already on the user’s device.

Another reason why Create ML is so popular is because of its ease-of-use. All you need to do with Create ML is have an extensive dataset (either text or image), write just a few lines of code, and run the playground! This is far more simpler than the other popular tools out there like Tensorflow and Caffe. Those tools require lots of code and don’t have a friendly visual interface. Create ML is all built into Xcode Playgrounds so you get the familiarity and best of all, it’s all done in Swift!

Prerequisites

In this tutorial, I will only be showing you how to create your own ML model using Create ML. If you would like to learn how to import a Core ML model into your iOS app, you can find the tutorial here.

At the time of writing, iOS 12 and macOS Mojave is still in beta. To successfully run the tutorial, you will need to be running macOS Mojave (10.14) and the Xcode 10 beta. Let’s get started!

The Image Classifier Model

The Data

We’ll first get started on building an image classifier model. We can add as many images with as many labels as we want, but for simplicity, we’ll be building an image classifier that recognizes fruits as apples or bananas. You can download the images here.

When you open the folder, you’ll notice two more folders: Training Data and Testing Data. Each folder consists of a mix between picture of apples and bananas. There are approximately 20 images of apples and 20 images of bananas in the folder called Testing Data and 80 images of apples and 80 images of bananas in Training Data. We will be using the images in Training Data to train our classifier and then use Testing Data to determine its accuracy.

If you want to build your own image classifier, it is important that you split your dataset into 80-20. Approximately 80% of your images go to Training Data and the remaining head to Testing Data. That way, your classifier has more data to train off of. In each of these folders, put the images in their respective folders. Name these folders based on the category label of the images.

Now, let’s open Xcode and click on Get Started with a Playground. When you do this, a new window opens up. This is the important part: under macOS, select the Blank template as shown below.

It is crucial that you select the Blank template under macOS and not iOS because the framework CreateML isn’t supported for iOS Playgrounds.

Name your playground and save it to wherever you want to. Let’s get coding now!

The Code

Now what I’m about to show you will blow your mind. All you need are 3 lines of code! Let me show you! Delete everything in the playground and type the following:

import CreateMLUI

let builder = MLImageClassifierBuilder()
builder.showInLiveView()

And that’s it! Make sure you enable the Live View feature in Xcode Playgrounds and you’ll be able to see the visual interface!

CreateMLUI is a framework just like CreateML but has a UI to it. As of now, CreateMLUI can only be used for image classification. Now, let’s see how we can interact with the UI! You’ll see it’s quite simple!

The User Interface

In the Live View, you’ll see that we need to drop images to begin! This is quite simple. Take the Training Data folder, and drop the entire folder into the area.

The moment you drop the folder, you’ll see the playground start to train the image classifier! In the console, you’ll see the number of images processed in what time and how much percentage of your data was trained!

This should take around 30 seconds (depending on your device). When everything is done processing, you should see something like this:

You’ll see a card with three labels: Training, Validation, and Evaluation. Training refers to the percentage of training data Xcode was successfully able to train. This should read 100%.

While training, Xcode distributes the training data into 80-20. After training 80% of training data, Xcode runs the classifier on the remaining 20%. This is what Validation refers to: the percentage of training images the classifier was able to get right. Usually, this can vary because Xcode may not always split the same data. In my case, Xcode had an 88% validation. I wouldn’t worry too much about this. Evaluation is empty because we did not give the classifier any testing data. Let’s do that now!

This should happen pretty quick. When everything is finished, your evaluation score should ready 100%. This means that the classifier labelled all the images correctly!

If you’re satisfied with your results, all that’s left is saving the file! Click on the arrow next to the Image Classifier title. A dropdown menu should appear displaying all the metadata. Change your metadata to how you would like it and save it to where you want to!

Open the CoreML model and view the metadata. It has everything you filled out! Congratulations! You are the author of your own Image Classifier model that’s super powerful, and takes only 17 KB!

You can import it into your iOS app and see how it runs! Next, let’s check out how to create our own Text Classifier. This requires a little more code!

The Text Classifier Model

The Data

Next, we’ll be building a Spam Detector model with Create ML. This is a type of model which determines if a message is either spam or ham (ham being not spam). Just like all machine learning applications, we’ll need some data. Download the sample JSON file here.

Opening it, you can see that it is a JSON table containing lots of messages, each labelled either spam or ham. The amount of data in this sample is very minimal compared to what you might want in your application.

The Code

Now, we have to ask Xcode to train the data. While we don’t have a nice and simple UI, the code we use is not too difficult. Type the following:

import CreateML
import Foundation
//1
let data = try MLDataTable(contentsOf: URL(fileURLWithPath: "/Users/Path/To/spam.json"))
let (trainingData, testingData) = data.randomSplit(by: 0.8, seed: 5)
let spamClassifier = try MLTextClassifier(trainingData: trainingData, textColumn: "text", labelColumn: "label")
//2
let trainingAccuracy = (1.0 - spamClassifier.trainingMetrics.classificationError) * 100
let validationAccuracy = (1.0 - spamClassifier.validationMetrics.classificationError) * 100
//3
let evaluationMetrics = spamClassifier.evaluation(on: testingData)
let evaluationAccuracy = (1.0 - evaluationMetrics.classificationError) * 100
//4
let metadata = MLModelMetadata(author: "Sai Kambampati", shortDescription: "A model trained to classify spam messages", version: "1.0")
try spamClassifier.write(to: URL(fileURLWithPath: "/Users/Path/To/Save/SpamDetector.mlmodel"), metadata: metadata)

Let me explain what happen. Most of the code should be fairly simple!

1. First, we create a constant called data which is a type of MLDataTable to our spam.json file. MLDataTable is a brand new object used to create a table determined to train or evaluate a ML model. We split our data into trainingData and testingData. Like before, the ratio is 80-20 and the seed is 5. The seed refers to where the classifier should start from. Then we define a MLTextClassifier called spamClassifier with our training data, defining what values of the data are text and what values are labels.
2. We create two variables called trainingAccuracy and validationAccuracy used to determine how accurate our classifier is. In the side pane, you’ll be able to see the percentage.
3. We also check how the evaluation performed. (Remember that the evaluation is the results used on text which the classifier has not seen before and how accurate it got them.)
4. Finally, we create some metadata for the ML model like the author, description, and version. We use the write() function to save the model to the location of our choice! In the image below, you’ll see that I chose the Desktop!

Run the playground. You can see the iterations in the console and the accuracy in the right hand bar! When all is done, the Core ML model is saved! You can view the model and see the metadata!

Tabular Classification

The Data

Tabular data is one of the most advanced and interesting features about Create ML. By observing a bunch of features in a table, Create ML can detect patterns and create a classifier to detect the target feature you want.

In this case, let’s deal with one of the most popular datasets in the world of machine learning- house pricings! And to make this more interesting, the dataset is not in the JSON format but rather the CSV format! Download the dataset here

This dataset is a modified version of the Boston housing dataset found on the UCI Machine Learning Repository. Opening the file you can see that there is a huge table filled with numbers and 4 abbreviations. Here’s what they mean:

• RM: The average number of rooms per dwelling
• LSTAT: The percentage of population considered lower status
• PTRATIO: The pupil-student ratio in town
• MEDV: The median value of owner-occupied homes

As you can guess, we’ll be using the 3 features (RM, LSTAT, PTRATIO) to calculate the final price (MEDV)!

The Code

Getting Xcode to read the table is quite simple! The following code should look really similar to the text classification code!

//1
let houseData = try MLDataTable(contentsOf: URL(fileURLWithPath: "/Users/Path/To/HouseData.csv"))
let (trainingCSVData, testCSVData) = houseData.randomSplit(by: 0.8, seed: 0)
//2
let pricer = try MLRegressor(trainingData: houseData, targetColumn: "MEDV")
//3
let csvMetadata = MLModelMetadata(author: "Sai Kambampati", shortDescription: "A model used to determine the price of a house based on some features.", version: "1.0")
try pricer.write(to: URL(fileURLWithPath: "/Users/Path/To/Write/HousePricer.mlmodel"), metadata: csvMetadata)

If you weren’t able to understand the above code, no problem! I’ll go through it step by step!

1. The first step is to reference our data in `HouseData.csv`. This is done through a simple call of `URL(fileURLWithPath:)`. Next, we define what portion of the data should be split into training and testing. We’ll split it into 80-20 like always and just to change things up a bit, let’s start from the beginning (setting the `seed` to 0).
2. Next, we define a type of regressor named `pricer` for our data using the brand new MLRegressor enumeration. This is one of the coolest parts about Create ML. There are a lot of regressors ML algorithms use: Linear, Boosted Tree, Decision Tree, and Random Forests. And these are just the most common ones. Unless you’re a ML expert, it can be hard to determine which one is the best suited for your data. This is where Create ML comes in to help. When you select `MLRegressor`, Create ML runs your data through all these regressors and chooses the best one for you. We choose the training data to be `houseData` and set the target column as `MEDV` which is the median price.

   Here’s some quick terminology. You may be wondering what’s the difference between a classifier and a regressor. A classifier groups the output of your data into classes or labels. Regressors, on the other hand, predicts the output value using training data. Regressors don’t have labels. Also, in machine learning, features are the variables in a dataset. In our case, the features are the average number of rooms, percent of population, and the pupil-student ratio. The target is also a column in our data which is what we want the regression to predict. In this case, it’s the median price of the houses.

3. Finally, we define some metadata for our model and save it to wherever we would like!

As of this writing, Create ML does not support showing the accuracy of MLRegressors. It can only show the maximum error and the square root error, both of which don’t contribute much to displaying how accurate the model is. Trust me though, when I say that the model Xcode generates is fairly accurate.

After the playground is done running, observe the right hand pane. It looks like Create ML has determined that Boosted Tree is the best regressor for our data! Isn’t that amazing? I have saved my Core ML model to my Desktop. Open your Core ML model and observe the metadata.

You can see that the model is a Pipeline Regressor and is about 10 KB. It takes in 3 features (just like we wanted) and output the final price!

Conclusion

In this tutorial, you saw how to create your own machine learning models using Apple’s newest framework Create ML! With just a few lines of code, you can create advanced, state-of-the-art machine learning algorithms to process your data and give you the results you want!

You saw how to train images, text, and tabular data in both CSV and JSON formats. With CreateMLUI it’s super simple to train images and while there is no UI for text and tabular data, you can write the code in less than 10 lines.

To learn more about Create ML, you can watch Apple’s video on Create ML from WWDC 2018 here. You can also check out Apple’s documentation on Create ML here.

You can download the final playground here. Along with the project, you’ll get access to the final Core ML models so you can see if your model matches up! Keep experimenting with Create ML and observe your results as you import them to your iOS app! Let me know how everything goes and share screenshots of your app using CoreML in the comments below!

Blockchain is one of the many disruptive technologies that has just started to gain traction among many people. Why? This is because blockchain is the founding technology for many cryptocurrencies like Bitcoin, Ethereum, and Litecoin. How exactly does Blockchain work though? In this tutorial, I’ll be covering everything there is to know about the blockchain technology itself and how to make your own “blockchain” in Swift. Let’s get started!

Blockchain: How does it work?

As the name implies, blockchain is a chain comprised of different blocks strung together. Each block contains 3 pieces of information: the data, a hash, and the previous block’s hash.

1. Data – Depending on the use, the data that is stored in a block depends on the type of blockchain. For example, in the Bitcoin blockchain, the data stored is the information relating to the transaction: the amount of money transferred and the information of the two people involved in the transaction.
2. Hash – You can think of a hash as a digital fingerprint. It is used to identify a block and its data. What’s important about hashes is that it’s a unique alphanumeric code, usually about 64 characters. When a block is created, so is its hash. When a block is modified, the hash is also modified. In this way, hashes are vital if you want to detect any changes made to the block.
3. Previous block’s hash – By storing the hash of the previous block, you can see how each block is linked to form a blockchain! This is what makes a blockchain so secure.

Take a look at this picture:

As you can see, each block consists of data (not shown), a hash, and the previous block’s hash. For example, the yellow block contains its own hash: H7S6, and the red block’s hash: 8SD9. This way, they all form a chain and are connected as such. Now, let’s say that a malicious hacker comes along and tries to modify the red block. Well remember, every time a block is modified in any way, the hash of that block changes! That way when the next block runs a check and sees that its previous hash does not match, it will not be accessible to the hacker since it effectively cut itself away from the chain.

This is what makes blockchain such so secure. It’s close to impossible to try to go back and change any data. While hashes do provide a great sense of secrecy and privacy, there are two more safeguards to keep blockchain even more secure: Proof-of-work and Smart Contracts. While I won’t be going into the details, you can read more about it over here.

The last way a blockchain secures itself is based on its location. Unlike most data which is stored on servers and databases, blockchain uses a peer-to-peer network (P2P). A P2P is a type of network that allows anyone to join and the data on that network is distributed to each recipient.

When someone joins this network, they get a full copy of the blockchain. When someone creates a new block, it is sent to everyone on the network. Then, through several complex programs, the node determines whether or not the block has been tampered with before adding it to the chain. That way, the information is available for everyone, everywhere. This may sound familiar if you’re a fan of HBO’s Silicon Valley. In that TV show, the protagonist uses a similar technology to create his new internet.

Since everyone has a copy of the blockchain, or nodes, they can form a consensus and determine which blocks are valid and which are not. Therefore, if you want to hack one block, you’ll have to hack more than 50% of the blocks on the network in order to pass along your information. This is why blockchain is perhaps one of the most secure technology created in the past decade.

About the Sample Application

Now that you have an understanding of how blockchain works, let’s get started with our sample application! Download the starter project here.

As you can see, we have two Bitcoin wallets. The first account, Account 1065, has 500 BTC available while the second account, 0217, has nothing. We send bitcoins to the other account using the send button. To earn BTC, we can press the Mine button which will give us a reward of 50 BTC. Basically what we are doing is observing the transaction that takes place between 2 Bitcoin accounts by looking at the console when the app is running.

You’ll notice that in the sidebar there are two important classes: Block and Blockchain. Opening these files, you’ll see that they’re empty. That’s because I’ll walk you through writing the logic for these classes. Let’s get started!

Defining the Block in Swift

Going over to Block.swift and let’s add the code which would define a Block. To begin, let’s break down what a Block is. We defined earlier that a block consists of 3 pieces of data: the hash, the actual data to be recorded, and the previous block’s hash. When we want to build our blockchain, we have to know whether the block is the first or the second. Well we can easily define this in Swift. Add to the class:

var hash: String!
var data: String!
var previousHash: String!
var index: Int!

Now, one last important code needs to be added. I mentioned earlier how every time a block is modified, its hash changes. This is one of the features of a blockchain which makes it secure. So we have to create a function which generates a hash filled with random letters and numbers. This function only requires a few lines of code:

func generateHash() -> String {
    return NSUUID().uuidString.replacingOccurrences(of: "-", with: "")
}

NSUUID is an object which represents a universally unique value that bridges to UUID. It’s built right into Swift and great for generating 32 character strings. This function generates a UUID, erases any hyphens, and returns the String, now known as the block’s hash. Block.swift should now look like this:

Now that we have defined the Block class, let’s define our Blockchain class. Switch over to Blockchain.swift to begin.

Defining the Blockchain in Swift

Just like before, let’s try to break down a blockchain into its fundamentals. In very basic terminology, a blockchain is nothing but a chain of blocks strung together, or in other words, a list with items held together. Does that sound familiar? If it does, that’s because it’s the definition of an Array! And this array is held together by blocks! Let’s add this to our code!

var chain = [Block]()

You’ll notice that inside the array is the Block class which we defined earlier. That’s all the variables we need for the blockchain. To finish up, we need to add two functions to our class. Try to answer this question based on what I taught earlier.

What are the two main functions in a blockchain?

I hope you were able to answer this question! The two main functions a blockchain has is to create the genesis block (the initial block) and to add a new block to its end. Now, of course, I’m not going into decentralizing the chain and adding smart contracts, but these are the basic functions! Add the following code to Blockchain.swift:

func createGenesisBlock(data:String) {
    let genesisBlock = Block()
    genesisBlock.hash = genesisBlock.generateHash()
    genesisBlock.data = data
    genesisBlock.previousHash = "0000"
    genesisBlock.index = 0
    chain.append(genesisBlock)
}

func createBlock(data:String) {
    let newBlock = Block()
    newBlock.hash = newBlock.generateHash()
    newBlock.data = data
    newBlock.previousHash = chain[chain.count-1].hash
    newBlock.index = chain.count
    chain.append(newBlock)
}

1. The first function we are adding is creating the genesis block. To do this, we create a function which takes in the data of a block as input. Then we define a variable named genesisBlock and make it of type Block. Since it is of type Block, it has all the variables and functions we defined earlier in Block.swift. We set the hash to generateHash() and its data to the input data. Sice it’s the first block, we set the previous block’s hash to 0000 in order to let us know that it’s the initial block. We set its index to 0 and we append it to the blockchain chain.
2. The next function we create is applicable for all the blocks after the genesisBlock and it creates the rest of the blocks. You’ll notice it’s very similar to the previous function. The only difference is that we set the previousHash to the previous block’s hash and by setting its index to its place in the blockchain.

And that’s all! We are done defining our Blockchain! Your code should look like something below!

Next, we’ll connect all the pieces to our ViewController.swift file and see it running in action!

The Wallet Backend

Switching over to ViewController.swift, we can see that all our outlets are connected. All we need to do is handle the transaction and print them to the console.

Before we do that however, we should explore the Bitcoin blockchain a little bit. Bitcoins come from an overall account, let’s say this account number was 0000. When you mine a BTC, it means that you solve math problems and are issued a certain number of bitcoins as a reward. This provides a smart way to issue the currency and also creates an incentive for more people to mine. In our app, let’s make the reward 100 BTC’s. First, let’s add the variables we need to our View Controller:

let firstAccount = 1065
let secondAccount = 0217
let bitcoinChain = Blockchain()
let reward = 100
var accounts: [String: Int] = ["0000": 10000000]
let invalidAlert = UIAlertController(title: "Invalid Transaction", message: "Please check the details of your transaction as we were unable to process this.", preferredStyle: .alert)

We defined an account with number 1065 and another account with number 0217. We also added a variable called bitcoinChain to be our blockchain and we let the reward be 100. We need a master account from where all the bitcoins come: this is our genesis account with number 0000. It has 10 million bitcoins. You can think of this account like a bank where for every reward, 100 bitcoins are taken out of it into the rightful account. We also define an alert which will be shown every time a transaction could not be completed.

Now, let’s code some generic functions which will run. Can you guess what these functions are?

1. The first function is for handling the transaction. We make sure that the sender and recipient accounts receive or deduct the right amount and this information is recorded into our blockchain.
2. The next function is for printing to the console the entire record- it will show each block and the data in each block.
3. The final function is for verifying whether the blockchain is valid by making sure the previous block’s hash matches with the information the next block has. Since we won’t be demonstrating any hacking methods, in our demo, the chain will always be valid.

The Transaction Function

Here’s the generic transaction function we have. Enter the following code right underneath the place where we defined our variables.

func transaction(from: String, to: String, amount: Int, type: String) {
    // 1
    if accounts[from] == nil {
        self.present(invalidAlert, animated: true, completion: nil)
        return
    } else if accounts[from]!-amount < 0 {
        self.present(invalidAlert, animated: true, completion: nil)
        return
    } else {
        accounts.updateValue(accounts[from]!-amount, forKey: from)
    }
    
    // 2
    if accounts[to] == nil {
        accounts.updateValue(amount, forKey: to)
    } else {
        accounts.updateValue(accounts[to]!+amount, forKey: to)
    }
    
    // 3
    if type == "genesis" {
        bitcoinChain.createGenesisBlock(data: "From: \(from); To: \(to); Amount: \(amount)BTC")
    } else if type == "normal" {
        bitcoinChain.createBlock(data: "From: \(from); To: \(to); Amount: \(amount)BTC")
    }
}

This may seem like a lot of code but at its core, it’s just defining some rules to follow for each transaction. At the top, we have 4 parameters for this function: to, from, amount, and type. To, From, and Amount are self-explanatory but Type is basically defining the types of transaction. There are 2 types: normal and genesis. A normal type of transaction would be between account 1065 and 0217 where as a genesis account would involve the account 0000.

1. The first if-else condition regards the from account. If it doesn’t exist or is short of money, we display the Invalid Transaction alert and return the function. Otherwise, we update the values as is.
2. The second if-else condition regards the account we send it to. If it doesn’t exist, then we leave it alone and return the function. Otherwise, we send the right amount of bitcoins to the account.
3. The third if-else statement deals with the type transaction. If there is a transaction involving a genesis block, we create a new genesis block, otherwise, we create a new block storing the data.

The Printing Function

At the end of every transaction, we want to see a list of all the transactions to make sure we know everything that is going on. Here’s what we type right underneath the transaction function.

func chainState() {
    for i in 0...bitcoinChain.chain.count-1 {
        print("\tBlock: \(bitcoinChain.chain[i].index!)\n\tHash: \(bitcoinChain.chain[i].hash!)\n\tPreviousHash: \(bitcoinChain.chain[i].previousHash!)\n\tData: \(bitcoinChain.chain[i].data!)")
    }
    redLabel.text = "Balance: \(accounts[String(describing: firstAccount)]!) BTC"
    blueLabel.text = "Balance: \(accounts[String(describing: secondAccount)]!) BTC"
    print(accounts)
    print(chainValidity())
}

This is a simple for loop where for every block in out bitcoinChain, we print the Block number, the hash, the previous block’s hash, and the data is stores. We update the labels on our UI so that they show the correct amount of BTC in each account. Finally, print a list of each account (which should be 3) and check the validity of the chain.

Now you should be getting an error on the last line of that function. This is because we haven’t defined our chainValidity() function yet so let’s get to it!

The Validity Function

Remember that a chain is valid if the previous block’s hash matches with what out current block says it is. We can easily to that with another for loop to iterate over every block:

func chainValidity() -> String {
    var isChainValid = true
    for i in 1...bitcoinChain.chain.count-1 {
        if bitcoinChain.chain[i].previousHash != bitcoinChain.chain[i-1].hash {
            isChainValid = false
        }
    }
    return "Chain is valid: \(isChainValid)\n"
}

Similar like before, we iterate over every block in our bitcoinChain and we check to see if the previous hash of a block matches with what our current block says it is.

And that’s it! We have defined our functions that will be used every time! Your ViewController.swift should now look like this:

All that’s left is to just link up the buttons to the functions. Let’s begin the final phase now!

Linking Everything Together

When our app first starts, we want the genesis account 0000 to send 50 BTC to our first account. From there, we’ll have the first account give 10 BTC to the second account. This can be done quite simply in 3 lines of code. Change your viewDidLoad function to look like this:

override func viewDidLoad() {
    super.viewDidLoad()
    transaction(from: "0000", to: "\(firstAccount)", amount: 50, type: "genesis")
    transaction(from: "\(firstAccount)", to: "\(secondAccount)", amount: 10, type: "normal")
    chainState()
    self.invalidAlert.addAction(UIAlertAction(title: "OK", style: .default, handler: nil))
}

We use the function which we defined earlier and we call the chainState() function at the end of it. Also, we add an OK action to our Invalid Transaction alert

Now let’s see what to add in the remaining four functions: redMine(), blueMine(), redSend(), and blueSend().

The Mining Functions

The mining functions are very easy and require only 3 lines of code. Here what to add:

@IBAction func redMine(_ sender: Any) {
    transaction(from: "0000", to: "\(firstAccount)", amount: 100, type: "normal")
    print("New block mined by: \(firstAccount)")
    chainState()
}
    
@IBAction func blueMine(_ sender: Any) {
    transaction(from: "0000", to: "\(secondAccount)", amount: 100, type: "normal")
    print("New block mined by: \(secondAccount)")
    chainState()
}

In the first mining function, we use our transaction function to send 100 BTC from the genesis account to our first account. We print that a block was mined and print the chainState. Similarly, we send 100 BTC to our second account in the blueMine function.

The Sending Functions

The sending functions are also slightly similar.

@IBAction func redSend(_ sender: Any) {
    if redAmount.text == "" {
        present(invalidAlert, animated: true, completion: nil)
    } else {
        transaction(from: "\(firstAccount)", to: "\(secondAccount)", amount: Int(redAmount.text!)!, type: "normal")
        print("\(redAmount.text!) BTC sent from \(firstAccount) to \(secondAccount)")
        chainState()
        redAmount.text = ""
    }
}
    
@IBAction func blueSend(_ sender: Any) {
    if blueAmount.text == "" {
        present(invalidAlert, animated: true, completion: nil)
    } else {
        transaction(from: "\(secondAccount)", to: "\(firstAccount)", amount: Int(blueAmount.text!)!, type: "normal")
        print("\(blueAmount.text!) BTC sent from \(secondAccount) to \(firstAccount)")
        chainState()
        blueAmount.text = ""
    }
}

First, we check to see if the redAmount or blueAmount text field is empty. If it is, we display the Invalid Transaction alert. If not, we’re clear to go. We use our transaction function to send money from the first account to the second account (or vice versa) with the amount entered and set the type to normal. We print how much has been send and we call our chainState() function. At the end, we clear the text field.

And we’re all done! Check to see if your code matches the image below!

Run the app and try it out! From the front end, it looks like a normal transaction app, but you’ll know what’s going on behind the scenes! Play around with the app transferring BTC from one account to another, try to trick it, and have fun with it!

Conclusion

In this tutorial, you learned how to create a blockchain in Swift and made your own Bitcoin transaction. Do note that in a real cryptocurrency’s backend, the implementation would look nothing like above as it would need to be decentralized with Smart Contracts but the above example is for learning purposes.

In this example, we used Blockchain for cryptocurrency but can you think of any other ways blockchain can be used? Let me know in the comments below! Hope you learned something new!

For reference, you can download the full project on GitHub.

Welcome to the first part of the Drag and Drop series! In this tutorial, you will learn how to implement the drag and drop functionality onto a UIViewController. In the next part of the series, you will learn how to use the Drag and Drop APIs with UITableViewControllers and UICollectionViewControllers .

One of the most anticipated releases of iOS 11 was the announcement of several new Drag and Drop APIs. So for those of you who aren’t familiar with Drag and Drop, it is a way to graphically move or copy data from one application to another or sometimes within the same application.

There are a lot of examples where you can implement Drag and Drop into your apps and while there are a numerous number of APIs for you to implement for different scenarios, it is really easy to implement. I’ll be teaching you how you can implement Drag and Drop within your apps specifically regarding to the UIViewController. Let’s get started!

Introduction to Drag and Drop

As mentioned earlier, Drag and Drop is a graphical way to move or copy data between two applications. Here’s some quick terminology:

1. Source App: The app from which the item or information is dragged (copied)
2. Destination App: The app from which the item or information is dropped (pasted)
3. Drag Activity: The dragging action, from start to finish.
4. Drag Session: The items which are dragged, managed by the system, throughout teh drag activity

A really neat fact about Drag and Drop is that while you are dragging items from one app to another, the source and destination apps continue processing as usual. Therefore, these two actions are synchronous. You can continue operating an app with another finger or start a new drag activity.

Also, there is no need to incorporate these APIs into UITextViews and UITextField as they automatically support drag and drop. You can configure UICollectionViews, UITableViews, and almost any view to support Drag and Drop.

For this tutorial, we’ll focus on adding Drag and Drop into UIViewControllers. Let’s dive into the implementation.

How to Implement Dropping

First, download the starter project over here. As you can see, we have a simple ViewController with two UIImageViews that are already linked to the code. These two image views are designed for you to drop an image. Now all we have to do is start coding!

Let’s start with integrating the Dropping APIs. We can now implement the API in several short steps. Our first step is to add the UIDropInteractionDelegate to the ViewController. So adjust the class to look like this:

class ViewController: UIViewController, UIDropInteractionDelegate {

    @IBOutlet weak var firstImageView: UIImageView!
    @IBOutlet weak var secondImageView: UIImageView!    
        
    override func viewDidLoad() {
        super.viewDidLoad()
        // Do any additional setup after loading the view, typically from a nib.
    }
    override func didReceiveMemoryWarning() {
        super.didReceiveMemoryWarning()
        // Dispose of any resources that can be recreated.
    }
}

Next, we have to add an interaction recognizer to our view. This can be implemented in our viewDidLoad method. After super.viewDidLoad(), type the following line:

view.addInteraction(UIDropInteraction(delegate: self))

Now, to make our class conform to the UIDropInteractionDelegate, we have to implement 3 methods. I will explain each one to you.

1. func dropInteraction(_ interaction: UIDropInteraction, performDrop session: UIDropSession)
   This method tells the delegate it can request the item provider data from the session’s drag items.
2. func dropInteraction(_ interaction: UIDropInteraction, sessionDidUpdate session: UIDropSession) -> UIDropProposal
   This method tells the delegate that the drop session has changed. In our case, we want to copy the items if the session has updated.
3. func dropInteraction(_ interaction: UIDropInteraction, canHandle session: UIDropSession) -> Bool
   This method check to see if the view can handle the session’s drag items. In our scenario, we want the view to accept images as the drag item.

Let’s first implement the second and third methods like this:

func dropInteraction(_ interaction: UIDropInteraction, sessionDidUpdate session: UIDropSession) -> UIDropProposal {
    return UIDropProposal(operation: .copy)
}

func dropInteraction(_ interaction: UIDropInteraction, canHandle session: UIDropSession) -> Bool {
    return session.canLoadObjects(ofClass: UIImage.self)
}

In the dropInteraction(_:sessionDidUpdate:) method, we return a UIDropProposal object and specify it’s a copy operation. You must return a UIDropProposal object if a view’s drop interaction delegate accepts dropped items. For the dropInteraction(_:canHandle:) method, the return value of the implementation indicates whether the drop is accepted. In the code above, we only accept drop activities that contain images.

We are almost done. We now have to add the code that will allow the app to perform the drop onto the view. Copy and paste the code below:

func dropInteraction(_ interaction: UIDropInteraction, performDrop session: UIDropSession) {
    // 1
    for dragItem in session.items {
        // 2
        dragItem.itemProvider.loadObject(ofClass: UIImage.self, completionHandler: { object, error in
            // 3
            guard error == nil else { return print("Failed to load our dragged item") }
            guard let draggedImage = object as? UIImage else { return }
            // 4
            DispatchQueue.main.async {
                let centerPoint = session.location(in: self.view)
                //5
                if session.location(in: self.view).y <= self.firstImageView.frame.maxY {
                    self.firstImageView.image = draggedImage
                    self.firstImageView.center = centerPoint
                } else {
                    self.secondImageView.image = draggedImage
                    self.secondImageView.center = centerPoint
                }
            }
        })
    }
}

1. Sometimes, when we drag an image onto our board, we may end up dragging more than one image. This returns an array session.items, so we run the following code for each dragItem.
2. We load the object which are UIImage in the dragItems.
3. In case of an error, we use a guard statement to handle the error. If an error exists (say, the item doesn’t conform to the UIImage class), then we print an error message.
4. If there is no error, we set the image of imageView to draggedImage.
5. We determine where the finger is placed in order to see if we set the image to the first imageView or the second imageView. Notice that the center of the image is exactly where our finger is.

Now, we are done! Let’s run the app and check to see how well it works! But, first make sure your code looks something like this:

Dragging and Dropping between apps is supported only on iPads. So if you want to drag and image from Safari or Photos and drop it onto the view, you will need to run the sample app on an iPad. iPhone only supports dragging and dropping within the app. On a simulator, it may take sometime for the image to copy itself onto the board.

If you follow me correctly, your app should run as expected! But what if you made a mistake and want to drag the image in the first image view into the second? Well it’s quite simple! Now, we have to implement the dragging API.

Dragging an item

Now, say that the image we had in our board was a mistake and we wanted it to be removed. Well, this would require us to drag the image to another location. To achieve this, we would need to implement the dragging APIs. Let’s see how it’s done!

First, let’s add the UIDragInteractionDelegate. To do this, we simply have to add UIDragInteractionDelegate to the list of protocols in our class:

class ViewController: UIViewController, UIDropInteractionDelegate, UIDragInteractionDelegate

Now you might get an error and that’s because we have not implemented the required protocol stubs in our code. Unlike the UIDropInteractionDelegate where we needed 3 methods to adhere to the protocol, we need only one in this case. Add this function at the end of the code:

func dragInteraction(_ interaction: UIDragInteraction, itemsForBeginning session: UIDragSession) -> [UIDragItem] {
}

This method is basically used for detecting the type of object being dragged and how to handle it.

Before we implement this method, we need to modify our viewDidLoad code slightly. Since we’ll be touching the image view, we need to enable the userInteraction property for both image views. Modify the viewDidLoad method to look like this:

override func viewDidLoad() {
    super.viewDidLoad()
    view.addInteraction(UIDropInteraction(delegate: self))
    view.addInteraction(UIDragInteraction(delegate: self))
    firstImageView.isUserInteractionEnabled = true
    secondImageView.isUserInteractionEnabled = true
}

Now, we’re almost done. Modify your dragInteraction(:_) function to this.

func dragInteraction(_ interaction: UIDragInteraction, itemsForBeginning session: UIDragSession) -> [UIDragItem] {
        if session.location(in: self.view).y <= self.firstImageView.frame.maxY {
            guard let image = firstImageView.image else { return [] }
            let provider = NSItemProvider(object: image)
            let item = UIDragItem(itemProvider: provider)
            return [item]
        } else {
            guard let image = secondImageView.image else { return [] }
            let provider = NSItemProvider(object: image)
            let item = UIDragItem(itemProvider: provider)
            return [item]
        }
}

And that’s all! Run your code and see if you can drag and drop between images from your Photos albums or the web. Copy them from the first image view to the second image view! It’s a drag fest! Check out the result below!

What’s Next

As you can see, it’s quite simple to add Drag and Drop to your apps enabling your app to activate a whole new set of powerful APIs. In this tutorial, you learned how to drag and drop images, but this can be applied to text as well! What’s more is that drag and drop in different view controllers such as UITableViews and UICollectionViews can really provide a seamless experience for your users.

To download the complete project, you can do so from the Github repository here.

To learn more about Drag and Drop, I reccomend checking out some of these video from WWDC 2017!

1. Introducing Drag and Drop
2. Mastering Drag and Drop
3. Drag and Drop with Collection and Table View
4. Data Delivery with Drag and Drop

Finally, here is Apple’s Official Documentation on Drag and Drop.

Let me know what you think of the tutorial and whether or not you would like to see a series on Drag and Drop!

There are several underused and not-so-popular frameworks hidden in the iOS SDK. Some of them can be useful and time-saving tools. The Natural Language Processing Class is one of them. Available in both Swift and Obj-C, the NSLinguisticTagger Class is used analyze natural language text to tag part of speech and lexical class, identify names, perform lemmatization, and determine the language and script. As a result, it is used extensively in machine learning programs. What does this really mean? Well, that’s what you’ll find out!

To begin, let’s go to Xcode and create a new playground. Name the playground whatever you want and set the platform to macOS. Once the playground is created, select everything and delete it. This way you’ll have a clean slate to work on. At the top of the playground, type the code below to import the following library.

import Foundation

To experiment with the new NLP API, let’s choose a big paragraph to mess around with. Here the block of text we’ll have our code analyze.

let quote = "Here's to the crazy ones. The misfits. The rebels. The troublemakers. The round pegs in the square holes. The ones who see things differently. They're not fond of rules. And they have no respect for the status quo. You can quote them, disagree with them, glorify or vilify them. About the only thing you can't do is ignore them. Because they change things. They push the human race forward. And while some may see them as the crazy ones, we see genius. Because the people who are crazy enough to think they can change the world, are the ones who do. - Steve Jobs (Founder of Apple Inc.)"

The very first thing we need to do is create a tagger. In Natural Language Processing, a tagger is basically a piece of software which can read text and “tag” various information to it such as part of speech, recognize names and languages, perform lemmatization, etc. We do this by calling the NSLinguisticTagger class. In the Playground file, insert the following lines of code:

let tagger = NSLinguisticTagger(tagSchemes:[.tokenType, .language, .lexicalClass, .nameType, .lemma], options: 0) 
let options: NSLinguisticTagger.Options = [.omitPunctuation, .omitWhitespace, .joinNames]

What are these tag schemes? Well, basically tag schemes are the constants used to identify the pieces of information we want from the text. The tag schemes we ask the tagger to look for are the token type, language, lexical class, name type, and lemma. We’ll be using these tag schemes in the rest of the tutorial. Here’s what each one is:

1. Token Type: A property which classifies each character as either a word, punctuation, or whitespace.
2. Language: Determines the language of the token
3. Lexical Class: A property which classifies each token according to its class. For example, it’ll determine the part of speech for a word, the type of punctuation for a punctuation, or the type of whitespace for a whitespace.
4. Name Type: This property looks for tokens which are part of a named entity. It’ll look for a personal name, an organizational name, and a place name.
5. Lemma: This basically returns the stem of a word token. I’ll be going into more detail about this later on.

The options portion basically tells the API how to split up the text. We’re asking the analyzer to ignore any punctuation and any whitespace. If there is a named entity, join it together.

With the initial setup, now we are ready to begin writing code using NLP in Swift! Before we continue to add any code, please make sure your code looks something like this.

Language Identification

So now, let’s begin by identifying what language this text is in. Obviously, we know that it’s in English but our computer doesn’t know that. Let’s create a function to determine the language:

func determineLanguage(for text: String) {
    tagger.string = text
    let language = tagger.dominantLanguage
    print("The language is \(language!)")
}

This code should be fairly simple to understand but in case you didn’t, don’t worry. I’ll break it down for you. We assign the string a user inputs to the tagger. We define a constant language to be the dominant language of the string the tagger is assigned to and print it.

Now let’s call the function with determineLanguage(for: quote). You should get an output which reads;

The language is en

Tokenization

The next step in parsing text is tokenization. Tokenization is the process of splitting sentences, paragraphs, or documents into your choice of length. In this scenario, we’ll be splitting the quote above into words. As before, let’s create a function:

func tokenizeText(for text: String) {
    tagger.string = text
    let range = NSRange(location: 0, length: text.utf16.count)
    tagger.enumerateTags(in: range, unit: .word, scheme: .tokenType, options: options) { tag, tokenRange, stop in
        let word = (text as NSString).substring(with: tokenRange)
        print(word)
    }
}

Let’s break down the code. Similar to what we’ve done earlier, we set the text a user inputs to be the tagger’s string. Next, we define a constant range to be the range of characters the API should tokenize. After that, we call the tagger.enumerateTags function to tokenize. We set the range, the length to .word, what Linguistic Tag Scheme to choose, and refer to the options constant we made earlier (i.e. what to ignore and what to join).

Upon every word the function tokenizes, we ask the function to print the word to the console. Now insert the following line of code to call the function:

tokenizeText(for: quote)

You should get a long list of all the words looking something like “Here, ‘s, to, the, … , Founder, of, Apple Inc.”

Here
's
to
the
crazy
ones
The
misfits
The
rebels
The
troublemakers
The
round
pegs
in
the
square
holes
The
ones
who
see
things
differently
They
're
not
fond
of
rules
And
they
have
no
respect
for
the
status
quo
You
can
quote
them
disagree
with
them
glorify
or
vilify
them
About
the
only
thing
you
ca
n't
do
is
ignore
them
Because
they
change
things
They
push
the
human
race
forward
And
while
some
may
see
them
as
the
crazy
ones
we
see
genius
Because
the
people
who
are
crazy
enough
to
think
they
can
change
the
world
are
the
ones
who
do
Steve Jobs
Founder
of
Apple Inc.

Lemmatization

Now that we have identified the language and dove in a little deeper by splitting up the quote into words, let’s go even more deeper by transforming the words into their base root. This is called Lemmatization. Take the word run for example. It can be transformed into running, ran, will run, etc. Since there are many forms of a word, Lemmatization breaks down the word into its most basic form.

Let’s implement the following function to lemmatize the words.

func lemmatization(for text: String) {
    tagger.string = text
    let range = NSRange(location:0, length: text.utf16.count)
    tagger.enumerateTags(in: range, unit: .word, scheme: .lemma, options: options) { tag, tokenRange, stop in
        if let lemma = tag?.rawValue {
            print(lemma)
        }
    }
}

This block of code is about 95% similar to our tokenizeText function. Instead of the .tokenType scheme, we use the .lemma scheme. Then, since the raw value of the tag is the lemma of the word, we have the function print to display the lemma for every word. Now invoke the function and take a look:

lemmatization(for: quote)

The list will look pretty similar to the same list you got after tokenizing the quote. But, there are a couple of differences. For example, notice how misfits, rebels, and troublemakers all have been return in their singular form. In the phrase “They are not fond of…”, see how are is returned to the console as be.

Parts of Speech

Diving in a little more deeper, let’s take every word in the quote and identify its part of speech.

func partsOfSpeech(for text: String) {
    tagger.string = text
    let range = NSRange(location: 0, length: text.utf16.count)
    tagger.enumerateTags(in: range, unit: .word, scheme: .lexicalClass, options: options) { tag, tokenRange, _ in
        if let tag = tag {
            let word = (text as NSString).substring(with: tokenRange)
            print("\(word): \(tag.rawValue)")
        }
    }
}

By now, the code should look really familiar. Same as our tokenizeText function, the only key difference is changing the scheme to .lexicalClass.

partsOfSpeech(for: quote)

The console returns each word and its corresponding part of speech. You can see the verbs, nouns, prepositions, adjectives, etc. Here are some of the results:

The: Determiner
troublemakers: Noun
The: Determiner
round: Noun
pegs: Noun
in: Preposition
the: Determiner
square: Adjective
holes: Noun
The: Determiner
ones: Noun
who: Pronoun
see: Verb

Named Entity Recognition

Finally, let’s see if the quote can recognize any names, organizations, or places in the quote above. Here’s the function below:

func namedEntityRecognition(for text: String) {
    tagger.string = text
    let range = NSRange(location: 0, length: text.utf16.count)
    let tags: [NSLinguisticTag] = [.personalName, .placeName, .organizationName]
    tagger.enumerateTags(in: range, unit: .word, scheme: .nameType, options: options) { tag, tokenRange, stop in
        if let tag = tag, tags.contains(tag) {
            let name = (text as NSString).substring(with: tokenRange)
            print("\(name): \(tag.rawValue)")
        }
    }
}

namedEntityRecognition(for: quote)

Notice how there’s one extra line of code? These are the tags we want our tagger to be on the lookout for. We want our tagger to list any personal names, place names, or organization names. Of course, change the scheme to .nameType and the rest should be straightforward.

Note: You’re probably wondering why it’s important to search for any named entities in the text. This is because it can lend a lot of insight into the context of the text.

As you probably expected, the function returns Steve Jobs as a Personal Name and Apple Inc. as an Organization Name.

Apple Inc.: Noun
Steve Jobs: PersonalName
Apple Inc.: OrganizationName

What’s Next

Hopefully by now you know how to use NLP in Swift. However, you’re probably wondering how exactly you can implement these in your apps. One way to implement the NLP API into your app is through the search result. Suppose you had a photos app with captions for each photo. One caption reads “Hike on Mt. Shasta”. Your user probably expect the same photo to show up when he/she search for hike, or hiking, or hikes. This can be implemented through lemmatization and that would definitely improve the user experience of your app.

For reference, you can refer to the complete Xcode playground on GitHub.

To learn more about NLP in Swift, you can check out Apple’s WWDC 2017 video here.

For more details about the NSLinguisticTagger class, you can refer to the official documentation here.