KDE Connect SMS: Nuremberg Megasprint Part 3

When interacting with other users of KDE Connect, I often notice something funny. People will often talk about how nice it is that they can control their computer from their phone (using the media controls, the mousepad plugin, or several of the others). For me, this has always been a secondary benefit. I use KDE Connect’s presentation plugin for all of my presentations but my primary goal has always been to be able to control my phone from my computer. I hate the phone keyboard, I hate the tiny screen, and I hate having to pull the thing out of my pocket.

On a daily basis, the number 1 thing I need to do with my phone is send or read SMS. In the United States, SMS is the de facto default communication system. In Europe, I basically assume that anyone I want to communicate with uses WhatsApp. In the US, with one friend I use WhatsApp, with another one friend I use Telegram, and with my family and other friends I use SMS. (Many people use Facebook Messenger but that is still not as widespread).

Those who have been very carefully following the KDE Connect channels might already know that we have been working on a desktop application which uses KDE Connect to load SMS conversation history and send SMS using the phone. (I have been keeping this under wraps because I know it is our #1 requested feature and I don’t want to tease anyone with a stream of “Oh yes, it’s almost there” half-promises)

The SMS app started March 2018 at the previous KDE Connect sprint. I arrived in Barcelona with a half-written contacts synchronization plugin and the goal to never touch my phone again. In only a few days, we had the contacts plugin in its current form and the skeleton of an SMS app (based on Qt’s QML chat app tutorial). It could read the display names and avatars from the synchronized contacts and you could use the compose message box to send SMS. There was no message history yet, just some statically-created items in the QML, but everything starts somewhere!

2018-03-26 KDE Connect v0-0.jpg
KDE Connect SMS v0.0

Around May, Aleix Pol and I were talking about what the interface should look like. We agreed that it should be very “normal”, like every other chat app, so I came up with this fabulous drawing.

1564053347781.jpg
Special thanks to PhotoScan for only being able to export photos with a watermark

The history of the SMS app has more details than we need for this post. Along the way, the KDE Connect Android app was updated with many content resolvers (and many StackOverflow references) to handle getting the SMS (and MMS) history. Several GUI elements have been “borrowed” from Kaidan. High on my soon-to-do list is move the Kaidan GUI elements which could be used by any other project looking to make a chat interface.

I am very happy with the current version of the app. I use it regularly and I find it very useful. Though it only has a few features, I have focused on fewer, more-stable features for the initial release. It can:

  • Show a list of conversation currently existing on the phone
  • Show conversation history
  • Send and display SMS
  • Display (not send) plain-text MMS including group messages
  • Understand non-text MMS
  • Update in real-time as more messages are received

I have left several thing for future releases:

  • Start a new conversation
    • I consider this a less-common usecase than replying to an existing conversation, so I focused on other things
  • Show MMS attachments
    • This requires upgrading the Android interface to read arbritrary attachments and upgrading the GUI to do something with those (like display pictures and offer downloads for everything else)
  • Send MMS
    • The current SMS-sending interface is very basic, so it will probably be significantly modified or even replaced.

Here is the current version of the SMS app:

2019-07-25 KDE Connect SMS v1.0

And the corresponding conversation on the phone:

Screenshot_20190725-135536

The SMS app now builds by default when you build KDE Connect so it will be officially released with the next release of KDE Connect. Hopefully I have time to iron out one or two more bugs before then!

Advertisements

KDE Connect mDNS: Nuremberg Megaspring Part 2

Sprints are a great time to talk in real-time to other project developers. One of the things we talked about at the KDE Connect part of the “Nuremberg Megasprint” was the problem that our current discovery protocol often doesn’t work, since many networks block the UDP broadcast we currently use. Additionally, we often get feature requests for more privacy-conscious modes of KDE Connect operation. Fixing either of these problems would require a new Link Provider (as we call it), and maybe we can fix both at once.

A New Backend

First, let’s talk about discovery. The current service discovery mechanism in KDE Connect is we send a multicast UDP packet to the current device’s /24 subnet. This is not ideal, since some networks are not /24, and since many public networks block packets of this sort. Alternatively, you can manually add an IP address which then establishes a direct connection. Manual connections work on many networks with block UDP, but it is a bit of a hassle. Can we find a better way to auto-discover services?

A few months ago, a user named rytilahti posted two patches to our Phabricator for KDE Connect service advertisement over mDNS (aka avahi, aka nsd, aka …). The patches were for advertisement-only (it still doesn’t establish a connection) but they were a good proof of concept to show that mDNS works on many institutional networks which block UDP multicast since mDNS is frequently used for other things like network printer discovery which are desired by those institutional networks.

I would post a screenshot here, but I don’t want to spread details of an internal network too far 🙂

At the sprint, we talked about whether we would like to move forward with these and we decided it was useful, so Albert Vaca and I put together two proof of concept patches to start trying to establish a connection using mDNS advertisements:

The patches are not yet fully working. We can both see each other and attempt to establish a connection but then something goes wrong and one of them crashes. Given that this was less than 8 hours of work, I would call this a success!

There is still plenty to do, but it was very helpful to be able to sit in-person and talk about what we wanted to accomplish and work out the details of the new protocol.

More Privacy

Before we talk about privacy, it helps to have a quick view of how KDE Connect currently establishes a connection:

  • As described above, both devices send a multicast UDP packet. This is what we call an “Identity Packet”, where each device send its name, capabilities (enabled plugins), and unique ID
  • If your device receives an identity packet from a device it recognizes, it establishes a secure TCP connection (if both devices open a connection, the duplicate connection is handled and closed)

As long as we are talking about a new backend, let’s think about ways to make KDE Connect more privacy-conscious. There are two problems to address:

  • Device names often contain personal information. For instances “Simon’s Phone” tells you that “Simon” is around
  • Device IDs are unique and unchanging. Even assuming I rename my phone, you can still track a particular device by checking for the same ID to show up again and again

Solving the first problem is easy. We want the user’s device name so we can display it in the list of available devices to pair with. So, instead of sending that information in the identity all the time, have some “discovery mode” switch which otherwise withholds the device name until a connection to an already-trusted device is established.

This leaves the second problem, which quite a bit more tricky. One answer is to have trusted user-selected trusted wifi networks, so KDE Connect doesn’t broadcast on a random wifi that the user connects to. But what if I connect to, say, my university network where I want to use KDE Connect but I don’t want to tell everyone that I’m here?

We don’t have a final answer to this question, but we discussed a few possible solutions. We would like some way of verifying ourselves to the other device which conceals our identity behind some shared secret, so the other device can trust that we are who we say we are, but other devices can’t fingerprint us. It is a tricky problem but not yet one to solve. Step 1 is to get the new mDNS backend working, step 2 is to add advanced features to it!

Welcome to KDE: Nuremberg Megasprint Part 1

Now that it has been over half a year since I started this blog, it is time to address one of the topics that I promised to address at the beginning: How I got started with KDE. I will do this in the context of the “Nuremberg Megasprint” which combined a KDE Connect sprint, a KDE Welcome / Onboarding sprint, and a KWin sprint.

At the Onboarding sprint, we were talking mostly about ways to make it easier for developers new to KDE to work on our software. Currently the path to getting that working is quite convoluted and pretty much requires that a developer read the documentation (which often doesn’t happen). We agreed that we would like the new developer experience to be easier. I don’t have a lot to say about that, but keep an eye on the Planet for an idea of what was actually worked on! Instead, since I am a relatively new KDE contributor, I will tell the story of how I got started.

I started using Plasma as a desktop environment around 2012, shortly after Ubuntu switched from Gnome 2, which I liked, to Unity, which I disliked. I tried playing with Mate and Cinnamon for Ubuntu, but I didn’t find either one was what I wanted. I had heard that KDE existed, but I didn’t know anything about it, so I gave it a try as well.

At the time, my main computer was an HP TM2-2000-series laptop, with a quite respectable 4GB RAM, some decent dual-core, first-generation Intel i5, and a little AMD GPU (which I could never get to work under Linux). But most importantly, it had a touchscreen with a capacitive digitizer for fingers, some styluses, or carrots (which usually work better than the special styluses) and a built-in Wacom digitizer for taking notes in class using the special pen.

HP-TM2-Walmart
An HP TM-2 Laptop, Almost in Tablet Mode

Plasma was nice to use on the touchscreen but most importantly, it had a built-in hotkey to disable the capacitive digitizer so I could write notes using the Wacom pen without my palm constantly messing everything up. It may sound silly, but that is literally the reason I started using KDE software!

Kubuntu came packaged with KDE Connect, which I was very excited by. Could I write SMS from the desktop without touching my phone and without installing questionable software? At the time, no. This was practically the first release of KDE Connect. It still had cool features, so I still loved it, but replying to SMS didn’t come until later.

Fast-forward the clock a couple of years. KDE Connect has had reply-to SMS features for awhile, but something was wrong. If you wrote a “long” SMS, KDE Connect would appear to accept it but then silently the message would never be sent. How curious! Since the only thing you could do was reply, it was hard to reproduce what was happening. I also noticed that KDE Connect had some work-in-progress, unreleased Telepathy plugin.

I started trying to set up Telepathy so that I would be able to send messages as well as just reply to them. I was able to get the plugin set up, which had (and still has, unfortunately) the very basic feature that you could enter a phone number and see messages sent and received in that “chat”, with no history or contacts-matching. Once I had the ability to send messages from KDE Connect, I noticed that any message I sent which was longer than 1 SMS (~140 bytes) would go missing.

At this point, the only software I had built was the Telepathy plugin (none of the core parts of KDE Connect). Luckily, the Android app is not difficult to build and debug. I followed the message I was trying to send through the app into an Android system call which was clearly for sending a single SMS (and apparently fails silently if the message is too long). I tweaked that part of the code to use the Android way of sending a multi-part SMS, posted the patch (to the mailing list, because I didn’t know Phabricator was the way to go since I hadn’t read the contributor documentation) and I have been hooked ever since.

Building the desktop app was more of a problem and is a better story to tell in the context of onboarding. I couldn’t figure out what cmake flags I needed. I am using Fedora, so I downloaded the source RPM to see if that would help me. I also couldn’t figure out how to read that, but I *did* figure out how to re-build the RPM with new sources. So, for about the first 8 months of my time in KDE, my workflow was:

  • Make a change
  • Rebuild the RPM (which took a relatively long time, even on my fairly fast computer)
  • Install the new RPM
  • Try to figure out why my change wasn’t working

Needless to say, this path was very cumbersome. Luckily, about this time, someone updated the KDE Connect wiki with the proper cmake flags to use!

After a certain amount of effort, I can now run KDE Connect in Eclipse, with the integrated debugger view (Note to readers: I recommend a different IDE for KDE/Qt development. Eclipse requires lots of manual configuration. Try KDevelop!)

2019-07-24 KDE Connect Eclipse
kdeconnectd, in Eclipse, paused in the debugger

That’s all for this post. I think it’s clear to say that my road to KDE development was far from straightforward. Hopefully we can make that road smoother in the future!

The Journey Begins

The mandatory first blog post. Best to start with a ramble.

I am 23 years old, but I do not have an Instagram. I see that lots of people my age do, and I am often invited to join them, but I am usually too hungry to photograph my food before eating it. Stopping to photograph a meal would take time away from working on all the projects I plan to blog about!

Software development is both my profession and my hobby. Some people like to do totally different things in their home life than their work life, like skiing or rock climbing or (food) photography. I do (most of) those things too, and I quite enjoy them. A certain percentage of my day is spent wondering if I could take better pictures if I bought a less-cheap smartphone. Then I remember that I don’t have anything I care to take pictures of, because I have other things to work on. The mood passes, and I keep happily using my cheap smartphone.

When I’m not wondering how my phone manages to mud of what could have been such a nice photo, I am often found working on software projects. Of particular interest right now is KDE Connect. KDE Connect is an application for synchronizing your connected devices, with the most focus on integrating your phone and your desktop. There are similar apps available, but in my eye KDE Connect combines the best of everything: Linux support, secure Device-to-Device communication – there is no way some 3rd party can intercept your data since it is never sent anywhere outside of your control – and, best of all, SMS support. Apparently the majority of consumers would prefer a skinny phone on which you have to struggle to type every word rather than have a physical keyboard, but that is not what I want. Ever since that market vanished and I had to switch to a one-piece candy-bar, I have dreamed of a way to never pull it out of my pocket. By working on KDE Connect, I can make that dream a reality.

Outside of traditional desktop programming languages, I often impress myself with how much I don’t know. While setting up this blog, I knew just enough to know what kind of layout I wanted. Responsive, so that if someone navigates to this blog – with much effort – on their keyboard-less phone it will look nice. Serious web development is a topic I have only briefly been exposed to. One day I would like to learn more, but that day will have to wait until I think of a sufficiently interesting project!

Hopefully the journey does not end here. I intend to write about my progress on KDE Connect so that those who do not want to dissect my often-inscrutable git commit messages can join in the experience. Here are some topics I intend to cover in the immediate future:

  • The secret project: Upgrades to SMS support I have been working on
  • How I got started with KDE

I am excited to see what this blog turns in to, and I appreciate you joining on the ride!

Good company in a journey makes the way seem shorter. — Izaak Walton

post