Kotlin & Gmail API - listing emails

The idea

Let's demonstrate the basic functionality on a useful idea. I have a folder/label in Gmail with a lot of emails. I want to list all email addresses - all senders.

Everything I need to do is a small console app to go through all emails with the given label and extracting the From header. If it contains the email address in format Name <email@address.com>, extract only the email address.

Let's dive into it!

Get credentials

Before we start, you need to create a new project in Google Cloud Console.

Under the new project, navigate to API & Services and enable Gmail API in Library.

In Credentials, click the + CREATE CREDENTIALS, select OAuth client ID and setup it like this:

Click Save and download the credentials for your newly created ID:

Save the downloaded file as credentials.json.

Kotlin project

Create a new Kotlin project with Gradle and add Google's dependencies:

dependencies {
    implementation "org.jetbrains.kotlin:kotlin-stdlib"
    implementation 'com.google.api-client:google-api-client:1.23.0'
    implementation 'com.google.oauth-client:google-oauth-client-jetty:1.23.0'
    implementation 'com.google.apis:google-api-services-gmail:v1-rev83-1.23.0'
    implementation 'org.jetbrains.kotlinx:kotlinx-coroutines-core:1.4.0-M1'
}

Authorization scopes

As we only want to go through labels, messages (we need just headers - metadata), we need a small and safe set of scopes:

private val SCOPES = setOf(
        GmailScopes.GMAIL_LABELS,
        GmailScopes.GMAIL_READONLY,
        GmailScopes.GMAIL_METADATA
)

Beware that if you provide the GmailScopes.GMAIL_METADATA, you are not able to access the whole message. You have to omit it if you want to get the message body.

Authorize with Gmail

Fortunately, Google libraries come with everything we may need including the server for receiving the authorization request. The whole implementation is as simple as:

private fun getCredentials(httpTransport: NetHttpTransport): Credential? {
    val inputStream = File("credentials.json").inputStream()
    val clientSecrets = GoogleClientSecrets.load(JSON_FACTORY, InputStreamReader(inputStream))
    val flow = GoogleAuthorizationCodeFlow.Builder(httpTransport, JSON_FACTORY, clientSecrets, SCOPES)
            .setDataStoreFactory(FileDataStoreFactory(File(TOKENS_DIRECTORY_PATH)))
            .setAccessType("offline")
            .build()
    val receiver = LocalServerReceiver.Builder().setPort(8888).build()
    return AuthorizationCodeInstalledApp(flow, receiver).authorize("user")
}

This code above outputs the request for authorization to the console:

Please open the following address in your browser:
  https://accounts.google.com/o/oauth2/auth?access_type=offline&client_id=...

Click the link, authorize the app, and the authorization token is received and stored in TOKENS_DIRECTORY_PATH. You only need to do this for the first time. Next time, the stored token is used.

Build client & get labels

We can now use the getCredentials() function above to build an authorized client, list all labels, and find the required one identified by labelName.

// Build a new authorized API client service.
val httpTransport = GoogleNetHttpTransport.newTrustedTransport()
val service = Gmail.Builder(httpTransport, JSON_FACTORY, getCredentials(httpTransport))
                .setApplicationName(APPLICATION_NAME)
                .build()

// Find the requested label
val user = "me"
val labelList = service.users().labels().list(user).execute()
val label = labelList.labels
        .find { it.name == labelName } ?: error("Label `$labelName` is unknown.")

List all emails

For listing all email messages, let's use a few of Kotlin's goodies - tailrec extension function with lambda as the last parameter.

We need to invoke the list request repeatedly until the nextPageToken is null, and doing so with tailrec is safer.

For each message, we invoke the process lambda to perform an actual operation.

private tailrec fun Gmail.processMessages(
    user: String,
    label: Label,
    nextPageToken: String? = null,
    process: (Message) -> Unit
) {

    val messages = users().messages().list(user).apply {
        labelIds = listOf(label.id)
        pageToken = nextPageToken
        includeSpamTrash = true
    }.execute()

    messages.messages.forEach { message ->
        process(message)
    }

    if (messages.nextPageToken != null) {
        processMessages(user, label, messages.nextPageToken, process)
    }

}

Process message

The code for listing emails above returns only id and threadId for each of the messages, so we need to fetch message details, extract From header, and eventually process it.

To speed up the process, let's use Kotlin's coroutines to perform the message fetching in parallel. First, introduce a custom dispatcher, so we can limit the number of threads.

private val MAX_FETCH_THREADS = Runtime.getRuntime().availableProcessors()

val executors = Executors.newFixedThreadPool(MAX_FETCH_THREADS)

val dispatcher = object : CoroutineDispatcher() {
    override fun dispatch(context: CoroutineContext, block: Runnable) {
        executors.execute(block)
    }
}

For extracting the email address from the Name <email@address.com> format, the code is simple:

private fun String.parseAddress(): String {
    return if (contains("<")) {
        substringAfter("<").substringBefore(">")
    } else {
        this
    }
}

Now, we can put things together. Of course, you should introduce some logic for catching exceptions, etc.

private fun Gmail.processFroms(
        user: String,
        label: Label,
        process: (String) -> Unit
) {
    runBlocking(dispatcher) {
        processMessages(user, label) { m ->
            launch {
                val message = users().messages().get(user, m.id).apply { format = "METADATA" }.execute()
                message.payload.headers.find { it.name == "From" }?.let { from ->
                    process(from.value.parseAddress())
                }
            }
        }
    }
}

Result

With all the code above, we can unique list of all senders like this:

val senders = mutableSetOf<String>()
service.processFroms(user, label) {
    senders += it
}

senders.forEach(::println)

Source code

The complete source code is available on Github.

---

Originally published on Localazy - the best developer-friendly solution for localization of web, desktop and mobile apps.