НСМЕП та Linux

НСМЕП: Національна система масових електронних платежів (National system of mass electronic payments).

Я розпочав дослідження цієї технології місяць тому, і запрошую вас до Яндекс.Клубу, в якому я пишу про прогрес аналізу цієї унікальної, але маловідомої широкому загалу системі.
Цілі дослідження - упевнитися в перспективності розробки НСМЕП для користувача, описати метод використання НСМЕП у середовищі Linux, та якнайповніше описати протокол взаємодії ПЗ з карткою (наразі, специфікації повністю закриті).

НСМЕП у Яндекс.Клубі: http://clubs.ya.ru/4611686018427461168/

Перрша ціль досягнута - вдалося запустити ПлатиМо-ІТ під wine: http://clubs.ya.ru/4611686018427461168/replies.xml?item_no=8

Encrypted Ubuntu One

Disclaimer

This is known to work in Oneiric (11.10), earlier versions don't work due to inotify bug.

This document is provided AS IS. The author assumes no liability in case any sort of data loss occurs, but will be very interested in hearing about such case. As always, before panicking, check the Trash directory. Since Natty Ubuntu One deletes files to Trash instead of wiping them from the file system.

About encryption

A lot of Ubuntu One users were asking about the security of Ubuntu One. The official answer is available at the Ubuntu One FAQ but you need to know that although the data is sent over encrypted connections between the servers and clients, it is not encrypted when stored. This allows you to publish the files, share them with other users and access the data via the REST API.

This may not be obvious but when you ask the installer to encrypt your /home directory all applications that are running within the context of your user (such as the browser, e-mail client, or any syncing software) have the access to unencrypted versions of the files. This protection comes handy when you lose the access to the computer hard drive (e.g. when stolen or HDD is sent out to be replaced/repaired) but anything you run under your account will work with decrypted files.

It is possible to use some sort of encrypted containers, such as TrueCrypt. The drawback of this method is that you will need to upload the whole container even if you have changed just one byte. Even if Ubuntu One supported delta uploads, changing some subset of data in a really secure container would have affected quite a large area, in order to conceal the actual location of the changed bytes.

Another way is to encrypt each file separately and upload the changes as they are happening by monitoring the files. This creates two copies of the files, one is encrypted and another is not. See this example for such solution involving REST API.

But I wanted to use the tools already available in Ubuntu. The aforementioned technology of /home directory encryption is based on eCryptFS - a layered encrypted file system implemented in the kernel itself. I wanted to be able to share some subset of files as well as keeping some of them private to my machines (while synced to the cloud)

Unfortunately, my first attempt to use eCryptFS-driven ~/.Private UDF (User Designated Folder) was not successful - the kernel was not sending inotify events (LP:773260) to the subscribers which meant that I needed to ask the synchronization software to rescan the files every time I modified the files. The kernel issue has been fixed but now Ubuntu One gets notified about changes in encrypted directories and it does not expect this type of notification (LP:872894)

Fortunately, a simple patch for Ubuntu One is sufficient to enable correct interaction with encrypted directories and I have created a PPA (ppa:rye/u1lp872894) which contains the forks of currently released Ubuntu One client with this patch applied. I will update this blog post when the fix is released to main distribution channels and will keep the PPA up to date as I am using it too.

Contrary to how it looks, eCryptFS does not use your account password to decrypt your data. In this case, you would have needed to re-encrypt all the files every time you'd change your password. Instead, your account password is used to encrypt and decrypt the real mount pass phrase, stored in ~/.ecryptfs/wrapped-passphrase.

You want to be able to decrypt your data later on (possibly) another machine, so you will need to know the pass phrase. When you set up private ecryptfs directory, you will be prompted for it or you can set it manually. In any case you will be able to decrypt it using ecryptfs-unwrap-passphrase application. Write the key down somewhere in a secure place - we will need that.

Setting it up

In order to set up a private directory, you will just need to run the following:

$ ecryptfs-setup-private
Enter your login passphrase: <type your account password>
Enter your mount passphrase [leave blank to generate one]: <just Enter>

************************************************************************
YOU SHOULD RECORD YOUR MOUNT PASSPHRASE AND STORE IT IN A SAFE LOCATION.
  ecryptfs-unwrap-passphrase ~/.ecryptfs/wrapped-passphrase
THIS WILL BE REQUIRED IF YOU NEED TO RECOVER YOUR DATA AT A LATER TIME.
************************************************************************

Done configuring.

Testing mount/write/umount/read...
Inserted auth tok with sig [de5cb8a79d96b6e4] into the user session keyring
Inserted auth tok with sig [87a5b4a8514fdcc9] into the user session keyring
Inserted auth tok with sig [de5cb8a79d96b6e4] into the user session keyring
Inserted auth tok with sig [87a5b4a8514fdcc9] into the user session keyring
Testing succeeded.

Logout, and log back in to begin using your encrypted directory.

So, now we have ~/.Private directory set up with a random pass phrase. Let's find out what the pass phrase really is (as the ALL CAPS warning above tells us):

$ ecryptfs-unwrap-passphrase
Passphrase: <type your account password>
d5da6e5eaf98d8f2b0a481995710b57e

The d5da6e5eaf98d8f2b0a481995710b57e string is our real mount pass phrase in this case. Now you are ready to ask Ubuntu One to enable syncing of ~/.Private folder:

$ u1sdtool --create-folder ~/.Private

Now verify that the folder has been created, make sure that you have ubuntuone-client version 2.0.0-0ubuntu2.2+lp872894 installed from the PPA and restart the client


$ u1sdtool --quit
$ u1sdtool --start

Put some file to your ~/Private folder - by default ecryptfs encrypts file names too, so when the sync completes, you will find something like ECRYPTFS_FNEK_ENCRYPTED.FXYl0wXfbLRRD-S7MDeYn8SYOWt.3exR-KbLnhYMpLY8ZwrgYH6xb8MBtBSjZQS9Hfvjo5BYEvrowNc- in ~/.Private folder and on the web interface at https://one.ubuntu.com/files/.

Multiple machines

OK, now when files are syncing, you can set up Private directory on another Ubuntu machine and you will be able to access the encrypted files created on another machine.

I have to say that this does not work extremely well when you edit the same file on both machines. eCryptFS seems to have some caching in place that prevents the updated lower file from being immediately available on another machine. If you use the machines sequentially (e.g. home machine and laptop you are carrying around) then this should not be a problem.

So, make sure that another machine is also running the version that has the fix for eCryptFS and set up the private folder:

$ ecryptfs-setup-private
Enter your login passphrase: <account password>
Enter your mount passphrase [leave blank to generate one]: <passphrase from first computer>
Enter your mount passphrase (again): <passphrase from first computer again>

************************************************************************
YOU SHOULD RECORD YOUR MOUNT PASSPHRASE AND STORE IT IN A SAFE LOCATION.
ecryptfs-unwrap-passphrase ~/.ecryptfs/wrapped-passphrase
THIS WILL BE REQUIRED IF YOU NEED TO RECOVER YOUR DATA AT A LATER TIME.
************************************************************************

Done configuring.

Testing mount/write/umount/read...
Inserted auth tok with sig [2c007b978eb65d67] into the user session keyring
Inserted auth tok with sig [b899c2e5fcb355f2] into the user session keyring
Inserted auth tok with sig [2c007b978eb65d67] into the user session keyring
Inserted auth tok with sig [b899c2e5fcb355f2] into the user session keyring
Testing succeeded.

Logout, and log back in to begin using your encrypted directory.

Good, now subscribe to ~/.Private folder using ubuntuone-control-panel or u1sdtool (--list-folders to find out folder-id followed by --subscribe-folder=folder-id).

Restart Ubuntu One client and you should start getting the files you put to first ~/.Private folder and when you visit ~/Private folder and you will be able to work with unencrypted files. You might need to refresh the ~/Private folder for the changes to be visible, this is a limitation of eCryptFS.

Windows support

To put it simple, this will not work with Windows, Android or any other Ubuntu One client for any OS other than Linux with eCryptFS support. Windows does not support eCryptFS mounts and there appears to be no plans for this. There may be different solutions with similar approach on Windows. If you happen to know one, feel free to leave the comment.

The web interface and Android file sync client will show encrypted file names and will download encrypted content. I am not aware of any eCryptFS-decrypting solutions for Android.

Thanks

After all, this blog post does not reveal anything new. The idea to use eCryptFS with Ubuntu One is quite common. The only value this long post adds is that it gives the address of the PPA. By the way, this blog post was originally written in reStructurizedText Markup in Vim.

Using mod_auth_openid with Ubuntu SSO

I have some internal web sites running on my home server. Earlier I configured WebAuth with WebKDC to create SSO-like experience. This was working well, but I wanted to use some 3rd party solution to be able to use auth on remote services even if local WebAuth host is not accessible.

The requirements were:

Specify the list of users which are allowed to login
Don't require server-side scripting

Since Ubuntu SSO is already used for Launchpad, Ubuntu One and various other services I decided that it is good enough for my small network.

Ubuntu hosts are using python-apache-openid python module and it has a very nice feature of being able to restrict access to some launchpad teams. It is not yet critical for me so I decided to look around in the archive and found libapache2-mod-auth-openid

This is a packaged version of mod_auth_openid from http://findingscience.com/mod_auth_openid/ - that web site contains the documentation for the module as well as examples.

The version in Oneiric is 0.5 which did not contain the feature I was after, the ability to restrict the access to some set of OpenID users without resorting to external script. So I went forward and updated the package to 0.6 and for the first time I was so pleased with the resulting package that I decided to send the updates to Debian maintainer. The updated package was successfully built in my ppa:rye/ppa and can be installed on Oneiric with

sudo apt-add-repository ppa:rye/ppa
sudo apt-get update
sudo apt-get install libapache2-mod-auth-openid

apt-add-repository is available from python-software-properties package.

Please verify that you have installed version 0.6:

$ apt-cache policy libapache2-mod-auth-openid
libapache2-mod-auth-openid:
  Installed: 0.6-0ubuntu1
  Candidate: 0.6-0ubuntu1

Configuration

I started with the following in my /etc/apache2/sites-available/default and marked the changed parts in bold

<VirtualHost *:80>
   ServerAdmin webmaster@localhost

   DocumentRoot /var/www
   <Directory />
     Options FollowSymLinks
     AllowOverride None
   </Directory>
   <Directory /var/www/>
     Options Indexes FollowSymLinks MultiViews
     AllowOverride None
     Order allow,deny
     allow from all
   </Directory>

  <Location />
    AuthType OpenID
    Require valid-user
    AuthOpenIDTrusted ^https://login.ubuntu.com/\+openid
    AuthOpenIDLoginPage /openid/
    AuthOpenIDCookiePath /
  </Location>

  <Location /openid>
    Order allow,deny
    Allow from all
    Satisfy any
  </Location>

  ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
  <Directory "/usr/lib/cgi-bin">
    AllowOverride None
    Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
    Order allow,deny
    Allow from all
  </Directory>

  ErrorLog ${APACHE_LOG_DIR}/error.log

  # Possible values include: debug, info, notice, warn, error, crit,
  # alert, emerg.
  LogLevel warn

  CustomLog ${APACHE_LOG_DIR}/access.log combined

  Alias /doc/ "/usr/share/doc/"
  <Directory "/usr/share/doc/">
    Options Indexes MultiViews FollowSymLinks
    AllowOverride None
    Order deny,allow
    Deny from all
    Allow from 127.0.0.0/255.0.0.0 ::1/128
  </Directory>

</VirtualHost>

I am using a custom login page since I don't need to enter my identifier every time. I am using a single provider so I am submitting an OpenID request automatically to Ubuntu SSO.

The /var/www/openid/index.html page is simple:

The query string parsing is actually needed to display the error that has been passed to the page in case the request is canceled on the provider or something else has happened.

Now the module needs to be enabled (this was not working in 0.5 out of the box) and apache needs to be restarted:

sudo a2enmod authopenid
sudo service apache2 restart

Now you have OpenID enabled for your virtual host root directory. The thing is it is enabled for everybody with Ubuntu SSO account so we need to restrict the access.

To find your OpenID identifier, log into your system using OpenID and browse the logs:

192.168.1.114 - https://login.ubuntu.com/+id/hPQWPsH ↩
  [12/Nov/2011:12:24:54 +0200] ↩
  "GET / HTTP/1.1" 304 210 "-" "Mozilla/5.0 ↩
  (Ubuntu; X11; Linux x86_64; rv:8.0)

Accounts that exist in launchpad can find out their +id/... value from the https://launchpad.net/~yournick, the openid.delegate will have this info. Launchpad login service and Ubuntu SSO service are currently using the same database, but we are targeting SSO.

<link rel="openid.delegate" href="https://login.launchpad.net/+id/hPQWPsH" />

The OpenID value needs to go to the apache config

  <Location />
    AuthType OpenID
    Require user https://login.ubuntu.com/+id/hPQWPsH
    AuthOpenIDTrusted ^https://login.ubuntu.com/\+openid
    AuthOpenIDLoginPage /openid/
    AuthOpenIDCookiePath /
  </Location>

Restart apache, and only the user specified in the Require user directive will be able to access the resource

Limitations

This module can be used only for ID authentication. While it is technically possible to request the OpenID provider to reply with user email or any other attributes, the module does not provide any protection from changing the values as they are traveling as a GET request back to the original form target page.

Switching to CIFS

I have a fileserver at home. It contains a backup of some photos, various Linux CD/DVD image, runs my virtual machines, serves as a Kerberos domain controller, printer server... well, pretty much everything I do has something to do with my home server.

Half a year ago I decided to move all my home directory to the server over NFS. I quickly configured NFSv4 with kerberos by following an extremely well-written community help article, created all the bind mounts and...

Well, I could transfer only half of my photos (6Gb) until the connection hung. I repeated this experiment for several more times and the worst part that any application that visited the mount point was unable to recover. I tried with various intr modes, various buffer sizes, TCP and UDP transports.

I swapped network cards, dropped Kerberos, switched to IPv4-only mode, switched to 100Mbps network instead of gigabit but it was still hanging.

I could transfer all my files over the same link using rsync over SSH, Apache WebDAV, and pretty much everything except NFS.

Today I gave up and set up Samba server on the same machine. And I could transfer everything I needed at the speed I expected without any issues.

I wanted to file the bug on Launchpad, but the kernel server runs at the kernel level, so it's not very comfortable for me to debug on a headless machine. And filing a bug w/o followup does not make sense. According to tcpdump, the server simply stopped answering at some point and according to my Google searches I am the only one experiencing this type of issue.

I am still keeping my NFS server running, but it feels a bit weird to use CIFS one a network of Linux-only machines.

Warning: Samsung ChatON

tl;dr version: Do not use ChatON on the public WiFi networks. The communication between client and server is not encrypted.

Update: Yesterday I found myself unable to sniff the packets properly but today plaintext messages are back, looks like there is something wrong with 46.137.191.242.

shortly after posting this (within an hour or so) my attempts to capture the plaintext messages started to fail, something has changed, continuing my investigation. Images are still going via HTTP though.

Two days ago Samsung launched their new IM service ChatON. ChatON Android application was released and its UI is definitely awesome. Having obtained my copy from Android Market I decided to check what protocol it is using for communication.

I launched tcpdump on my (rooted) Acer Liquid and started listening for the messages.

Among the lines of https (encrypted) messages flowing back and forth I found the following:

21:05:01.146225 IP 46.203.98.114.42914 > 46.137.191.242.5223: P 118:345(227) ack 1 win 5600 
 0x0000:  4500 0117 30e8 4000 4006 8940 2ecb 6272  E...0�@.@..@.�br
 0x0010:  2e89 bff2 a7a2 1467 ab45 ae43 4473 fdf8  ..����.g�E�CDs�
 0x0020:  8018 15e0 fab9 0000 0101 080a 00db c787  ...���.......��.
 0x0030:  0ba8 7d12 6264 3530 6263 3337 2d65 6532  .�}.ba50bc37-ee2
 0x0040:  382d 3432 6366 2d61 6531 352d 3133 3731  8-42cf-ae15-1371
 0x0050:  3334 3466 3363 3464 0104 00bb 08cd f7ef  344f3c4d...�.��
 0x0060:  87d8 e820 1000 1800 2248 3462 3064 6365  .��....."H4b0dce
 0x0070:  6638 2d64 3733 322d 3464 3038 2d62 3062  f8-d732-4d08-b0b
 0x0080:  612d 6536 3862 3263 3931 3066 6662 6264  a-e68b2c910ffbbd
 0x0090:  3530 6263 3337 2d65 6532 382d 3432 6366  50bc37-ee28-42cf
 0x00a0:  2d61 6531 352d 3133 3731 3334 3466 3363  -ae15-1371344f3c
 0x00b0:  3464 2a0c 3338 3039 3133 3532 3930 3539  4d*.380913529059
 0x00c0:  320c 3338 3039 3337 3532 3539 3836 3a24  2.380937525986:$
 0x00d0:  6264 3530 6263 3337 2d65 6532 382d 3432  bd50bc37-ee28-42
 0x00e0:  6366 2d61 6531 352d 3133 3731 3334 3466  cf-ae15-1371344f
 0x00f0:  3363 3464 420f 3335 3335 3039 3033 3132  3c4dB.3535090312
 0x0100:  3536 3238 354a 1068 692c 2068 6f77 2061  56285J.hi,.how.a
 0x0110:  7265 2079 6f75 3f                        re.you?

21:05:07.236157 IP 46.137.191.242.5223 > 46.203.98.114.42914: P 228:422(194) ack 345 win 62 
 0x0000:  4500 00f6 af74 4000 3206 18d5 2e89 bff2  E..��t@.2..�..�
 0x0010:  2ecb 6272 1467 a7a2 4473 fedb ab45 af26  .�br.g��Ds�۫E�&
 0x0020:  8018 003e b6d2 0000 0101 080a 0ba8 8550  ...>��.......�.P
 0x0030:  00db c787 6264 3530 6263 3337 2d65 6532  .��.ba50bc37-ee2
 0x0040:  382d 3432 6366 2d61 6531 352d 3133 3731  8-42cf-ae15-1371
 0x0050:  3334 3466 3363 3464 0106 009a 0a48 3462  344f3c4d.....H4b
 0x0060:  3064 6365 6638 2d64 3733 322d 3464 3038  0dcef8-d732-4d08
 0x0070:  2d62 3062 612d 6536 3862 3263 3931 3066  -b0ba-e68b2c910f
 0x0080:  6662 6264 3530 6263 3337 2d65 6532 382d  fbbd50bc37-ee28-
 0x0090:  3432 6366 2d61 6531 352d 3133 3731 3334  42cf-ae15-137134
 0x00a0:  3466 3363 3464 1000 1a4a 0a0c 3338 3039  4f3c4d...J..3809
 0x00b0:  3337 3532 3539 3836 120c 3338 3039 3133  37525986..380913
 0x00c0:  3532 3930 3539 1884 d185 fb9e c816 221b  529059..�.�.�.".
 0x00d0:  6920 7761 6e74 2061 205b 686d 5d20 616e  i.want.a.[hm].an
 0x00e0:  6420 6120 5b35 286c 6c29 5d28 c3e5 bb98  d.a.[5(ll)](���.
 0x00f0:  b126 3000 2800                           �&0.(.

Do you see something?

Here's what you see:

Basically, connection to the server is not encrypted

The client is using Google Protobuf protocol to send messages back and forth between client and server and the communication is not encrypted in any way. After more careful examination I found that the session initiation IS encrypted, so that it may not be possible to find whom exactly with you are talking to but all the messages will be visible to everybody around you if you are using an unencrypted open network such as you can find at the local cafe shops, restaurants, shopping centres etc.

You may not be in that amount of danger if you are using your cell phone carrier for the internet connection, that communication is encrypted between your phone and the cell towers, however you might not always realize that you have switched to open WiFi network and keep using ChatON.

I don't have a Bada-powered device nearby to verify whether that uses the same servers so I will assume that the same unencrypted protocol is used on all Bada devices and Samsung featurephones that have ChatON installed unless I have the proof that it is doing otherwise.

Interesting geeky detail - the servers are running on port 5223 which is usually associated with XMPP over SSL but it is actually a proprietary protocol (well, based on Google's protobuf). The chat servers are running on Amazon AWS hosts.

File uploads are also running over plain HTTP/1.1 without encryption:

POST /file?uid=bd50bc37-...-1371344f3c4d¶m=7daffaa462b802b...92e37870 HTTP/1.1
content-type: image/jpeg
content-length: 48401
User-Agent: Dalvik/1.2.0 (Linux; U; Android 2.2.2; Liquid Build/FRG83G)
Host: eu.file.samsungchaton.com
Connection: Keep-Alive

...

GET /4b/0d/ce/f8/d7/32/4d/08/b0/ba/e6/8b/2c/91/0f/fb/4b0dcef8-d732-4d08-b0ba-e68b2c910ffb/
       1318879486269_239.jpg?AWSAccessKeyId=AKIAIXENATYOW4T2DJSQ&
       Expires=1319052296&Signature=6UFD%2FYS9Vlls9X7WJov8GcH7EGs%3D HTTP/1.1
User-Agent: Dalvik/1.2.0 (Linux; U; Android 2.2.2; Liquid Build/FRG83G)
Host: eu.chaton-file.s3.amazonaws.com
Connection: Keep-Alive


HTTP/1.1 200 OK
x-amz-id-2: PNglgPsPAkR7SvhFPHk2bkl901Q6MGedePoaCRf/RGArSM36lZgtkMWLN10nmzZK
x-amz-request-id: E25D27220B372F64
Date: Mon, 17 Oct 2011 19:24:58 GMT
Last-Modified: Mon, 17 Oct 2011 19:24:51 GMT
ETag: "3ec7437168455698b4367ed303bfcfad"
Accept-Ranges: bytes
Content-Type: image/jpeg
Content-Length: 738
Server: AmazonS3

...

Yeah.

Ah, when user is not properly registered (ChatON on Android Emulator) the message returned is...

13:56:58.562553 IP 46.203.53.202.59299 > 46.137.191.242.5223: P 216:450(234) ack 185 win 3456 
 0x0000:  4500 011e df2c 4000 4006 079d 2ecb 35ca  E...�,@.@....�5
 0x0010:  2e89 bff2 e7a3 1467 38a3 84e8 4513 9da9  ..����.g8�.�E..�
 0x0020:  8018 0d80 2c9f 0000 0101 080a 001a 66c4  ....,.........f
 0x0030:  0c05 21a6 6264 3530 6263 3337 2d65 6532  ..!�bd50bc37-ee2
 0x0040:  382d 3432 6366 2d61 6531 352d 3133 3731  8-42cf-ae15-1371
 0x0050:  3334 3466 3363 3464 0104 00c2 088a e2a2  344f3c4d...�..
 0x0060:  fea5 8004 1000 1800 2248 6264 3530 6263  ��......"Hbd50bc
 0x0070:  3337 2d65 6532 382d 3432 6366 2d61 6531  37-ee28-42cf-ae1
 0x0080:  352d 3133 3731 3334 3466 3363 3464 6366  5-1371344f3c4dcf
 0x0090:  6662 3933 3939 2d63 3935 332d 3462 6465  fb9399-c953-4bde
 0x00a0:  2d39 3266 352d 6639 6435 3136 3862 3562  -92f5-f9d5168b5b
 0x00b0:  3764 2a0c 3338 3039 3133 3532 3930 3539  7d*.380913529059
 0x00c0:  320c 3338 3036 3336 3137 3038 3335 3a24  2.380636170835:$
 0x00d0:  6264 3530 6263 3337 2d65 6532 382d 3432  bd50bc37-ee28-42
 0x00e0:  6366 2d61 6531 352d 3133 3731 3334 3466  cf-ae15-1371344f
 0x00f0:  3363 3464 420f 3335 3335 3039 3033 3132  3c4dB.3535090312
 0x0100:  3536 3238 354a 174e 6f77 2069 7420 6973  56285J.Now.it.is
 0x0110:  2077 6f72 6b69 6e67 2061 6761 696e       .working.again
13:56:59.643944 IP 46.137.191.242.5223 > 46.203.53.202.59299: P 185:369(184) ack 450 win 62 
 0x0000:  4500 00ec 8237 4000 3206 72c4 2e89 bff2  E..�.7@.2.r�..�
 0x0010:  2ecb 35ca 1467 e7a3 4513 9da9 38a3 85d2  .�5�.g��E..�8�.
 0x0020:  8018 003e a34b 0000 0101 080a 0c05 2a79  ...>�K........*y
 0x0030:  001a 66c4 6264 3530 6263 3337 2d65 6532  ..f�bd50bc37-ee2
 0x0040:  382d 3432 6366 2d61 6531 352d 3133 3731  8-42cf-ae15-1371
 0x0050:  3334 3466 3363 3464 0105 0090 088a e2a2  344f3c4d......
 0x0060:  fea5 8004 1248 6264 3530 6263 3337 2d65  ��...Hbd50bc37-e
 0x0070:  6532 382d 3432 6366 2d61 6531 352d 3133  e28-42cf-ae15-13
 0x0080:  3731 3334 3466 3363 3464 6366 6662 3933  71344f3c4dcffb93
 0x0090:  3939 2d63 3935 332d 3462 6465 2d39 3266  99-c953-4bde-92f
 0x00a0:  352d 6639 6435 3136 3862 3562 3764 1a13  5-f9d5168b5b7d..
 0x00b0:  0a0e 3436 2e31 3337 2e31 3137 2e32 3334  ..46.137.117.234
 0x00c0:  10e7 2820 96cc b5b5 b126 2a20 08f2 2e12  .�(..̵��&*..�..
 0x00d0:  1b4d 6573 7361 6765 2052 6563 6569 7665  .Message.Receive
 0x00e0:  7220 6973 2049 6e76 616c 6964            r.is.Invalid

Ubuntu One, Headless

Hello again!

Notice: To run real Ubuntu One syncdaemon in headless mode, see Ubuntu One Headless wiki page.

Another idea I have been thinking about for quite a long time but it became extremely simple after Ubuntu One introduced REST API for file access. This API is used by a really wonderful Ubuntu One Files application for Android by Michał Karnicki and Web&Mobile team and John's u1rest library.
I decided to see how hard would it be to create a command line application similar to ftp to upload and download files and navigate my cloud storage using the API directly. The application needed to be usable in the server environment too.

This is an initial version:

rtg@lucidity:~$ ./ubuntuone-rest-files-client.py --oauth a:b:c:d
Welcome to Ubuntu One!
User id: 52053, name: Roman
Usage: 26.1 GiB/65.0 GiB
> ls
/~/Pictures                                        - -
/~/.gpass                                          - -
/~/Music                                           - -
/~/Pictures - Liquid                               - -
/~/Backup                                          - -
/~/Public                                          - -
/~/Documents                                       - -
/~/.ubuntuone/Purchased from Ubuntu One            - -
/~/Videos                                          - -
> cd ~/Documents
/~/Documents&amp;gt; ls
Articles/                                             2011-04-05T20:07:40Z
Certificates/                                         2011-04-18T07:32:28Z
CV/                                                   2010-12-01T11:14:08Z
DVD Labels.odg                                  13332 2011-03-06T20:38:50Z
eBooks/                                               2011-08-05T15:08:03Z
_image_vab-document.png                        145336 2011-04-30T10:15:33Z
...
/~/Documents> get _image_vab-document.png
Downloading 145336 bytes to _image_vab-document.png...  Done
/~/Documents> cd ../Ubuntu\ One
/~/Ubuntu One> put desktopcouch_0.6.4.orig.tar.gz
Uploading desktopcouch_0.6.4.orig.tar.gz to https://files.one.ubuntu.com/content/~/Ubuntu%20One/desktopcouch_0.6.4.orig.tar.gz
Content size: 105609
/~/Ubuntu One>

Well, you understand the idea. Please understand that at the moment this is a very rough version, so download and use it if you really like to play with something that may suddenly break.

Where to get and how to use

You will need 2 files from ubuntuone-scripts repository - bzr branch lp:~rye/+junk/ubuntuone-scripts, you can download the latest versions directly from ubuntuone-sso-login.py and ubuntuone-rest-files-client.py.
In order to get the a:b:c:d values that are the value for --oauth option run the first script as follows:

$ ./ubuntuone-sso-login.py
Creating new entry for buzz
SSO login: Your Ubuntu One SSO e-mail
password: Your password
Using SSO URL: https://login.ubuntu.com/api/1.0/authentications?ws.op=authenticate&amp;amp;token_name=%22Ubuntu+One+%40+buzz%22
OAuth info:
a:b:c:d
Ping result: ok 1/7

We are interested in the line after OAuth info. It will be a long string.

Warning! This OAuth string should be treated as secret, since it is composed of your OAuth consumer key:consumer secret:token:token secret. This string enables anybody who knows it to access your files, and CouchDB databases, think of it as a login/password pair that you can remove when needed. If you suspect that somebody else has that string, go to Ubuntu One web site and remove the corresponding entry. Then you can run ubuntuone-sso-login.py again and get a new set of credentials for the script.

Then run ubuntuone-rest-files-client.py:

./ubuntuone-rest-files-client.py --oauth a:b:c:d
Welcome to Ubuntu One!
User id: 52053, name: Roman
Usage: 26.1 GiB/65.0 GiB
>

That's pretty much it. The following commands are supported:

get remote [local] - download the remote file as local, in case local name is omitted it will use the original name
put local [remote] - upload the local file, same rules for omitting the remote name
mget remote1 [remote2 remote3 ... remoteN] - download the files to the current working directory
cd folder - change remote directory. Please remember to quote the path if it contains spaces or escape them - "/~/Ubuntu One" or /~/Ubuntu\ One
quit or Ctrl+D - terminate the script
ls - list folder contents, in case some file is published the URL will be printed
publish remote - publish an already uploaded file. Will print public URL
unpublish remote - take down the published file

More commands will be added later, at the moment this is a working proof of concept but I am very happy with the API so far. I will make the script much more stable and use less resources in the future (at the moment the whole file is read into memory during upload/download - thank you httplib2).

Ah, by the way, it works on Fedora too: Fedora 14 screenshot

Проблемы Acer Liquid E

Так как этот телефон все еще продается в магазинах, то хотел бы описать его недостатки, которые становятся заметными через некоторое время. Возможно, это кому-нибудь поможет в выборе.

Перезагрузка во время звонка

В GSM сети life:) Украина с тремя разными SIM картами телефон неоднократно перезагружался при входящем звонке, если во время приема звонка шла передача данных по GPRS/EDGE. На видео выше прошивка 2.1 - снял bigzverua, на моем экземпляре прошивка 2.2 (даже 2.2.2, baseband A1-05.01.06) и повторить можно без проблем. На SIM Beeline и Utel (в режиме GSM роуминга в Kyivstar и в их UMTS сети) таких проблем не наблюдалось. Также у телефона может зависнуть GSM часть при исходящем звонке. В таком случае в трубке будет слышен тихий шум, через некоторое время пропадет GSM сигнал и телефон перезагрузится.

Датирование фотографий - 2002-12-08

Все фотографии, сделанные стандартным приложением Camera на Acer Liquid E будут датированы восьмым декабря 2002 года в полдень. Эта дата прошита в библиотеке, которая работает с камерой. В галерее фотографии будут группироваться правильно до первого подключения телефона в режиме USB носителя к компьютеру. После отключения от компьютера запускается сканер медиаконтента, который перечитает дату создания снимка и все фотографии будут отображаться в группе 2002-го года. При загрузке фотографий на ресурсы, которые умеют смотреть на EXIF (или импорт в десктопные приложения), сортировка по дате будет бесполезна. Пример.

Температура

Очень часто при работе в сети нижняя часть телефона ощутимо греется (больше +38℃). Этого не происходит при использовании телефона как плеера в кармане. Недавно пользовались телефоном как GPS навигатором (Google Maps) в машине с подключенной зарядкой. Телефон разогревался до такого состояния, что отключал зарядку аккумулятора.

Элементы управления

На телефоне кнопка включения и камеры расположены так, что сложно сразу определить перевернут ли телефон (например, нажать на кнопку питания для разблокировки экрана до того, как посмотреть на аппарат). Сенсорная клавиша "Меню" достаточно часто нажимается при попытке ввода текста в портретной ориентации.
Чувствительность экрана также оставляет желать лучшего. После того, как я снял защитную пленку (есть в комплекте), которую я заботливо наклеил в первый день после покупки, экран стал ощутимо чувствительнее.

Acer UI

В этой модели панель нотификаций в Acer UI находится снизу, поэтому постоянно нажимается при попытке набрать пробел на экранной клавиатуре. В следующих моделях панель нотификаций вернули на место. Acer UI появился в прошивке 2.2, его можно выключить и использовать почти стандартный интерфейс Android.
К сожалению, пользоваться клавиатурой XT9 также не совсем удобно, отсутствуют некоторые специальные символы, нет украинских букв. Клавиатуру можно поставить другую из Android Market.

Место для приложений

Для пользовательских данных существует раздел /data на 200 мегабайт. Даже если приложения ставятся на SD-карту, odex файл с кодом приложения копируется в /data/dalvik-cache, и при больших объемах приложений, внутренней памяти начинает не хватать. Отваливается возможность получать SMS или сохранять кеш браузера.

Аппарат остается моим основным телефоном, но уже с Utel (если пришлось менять оператора из-за перезагрузок, то почему бы не на UMTS?). Даты правлю с помощью скриптов при импортировании а от Acer UI избавился, поставив прошивку от Vodafone Italy (которая, впрочем, перезагрузки с life:) не исправила). Если бы я знал все, что я описал выше до покупки, то этот телефон бы брать не стал. Хотя, может это был просто намек, что надо менять оператора?

Проблема с HUAWEI E1550 после активации голоса через MTS PatchDataCard

Update: Еще раз перешил модем, уже поправив дату для программы и убрав невозможность разблокирования. В принципе, это активатор голоса для life:) украина, 255-06, требуется только добавить свой IMEI. Скачать можно с Ubuntu One или Яндекс.Народа.

Загорелся я идеей подключить свой life:) модем еще и для голоса. Первый вариант, dc-unlocker пока был отложен в сторонку т.к. я еще не готов был платить за вещь, которая, возможно, мне не будет нужна.

После того, как я произвел процедуры, описанные в активации голоса на модемах HUAWEI E1550, E1750, Е156, E160 и подобных, мой модем больше не захотел подключаться к сети life:) и настаивал на переезде в Россию к МТС. AT^CARDLOCK возвращали "+CME ERROR: 16". Похоже, что в патче к nvram SIM LOCK сделался неснимаемым для моего модема.

В Resource Hacker, который будет в том-же архиве по инструкции нужно сменить IMEI. Но для того, чтобы все заработало мне пришлось менять и SIMLOCK_RANGE_0 на значения life - 25506. На всякий случай оставил 25506|25506|2 - для чего еще раз идентификатор и что значит "2" - я не знаю.Возможно это и есть флаг, который указывает на то, можно ли снимать привязку. После изменения любого значения в ресурсе, программа будет ругаться на несоответствие CRC. Над чем именно вычисляется CRC я не докопался, поэтому пошел в лоб - отлавливать значение CRC, которое программа хочет увидеть.

После скачивания в IDA ставим breakpoint на text.00403853 (в районе CRCSlow), запускаем программу через отладчик, когда сработает breakpoint в регистре ESI будет значение контрольной суммы, которое нужно прописать в [CRC] VALUE. Переводим значение из шестнадцатеричного в десятичный, еще раз идем в Resource Hacker, меняем значение суммы, Compile Script, Save и запускаем PatchDataCard снова.

Фух. Я думал, что придется возиться долго.

Кстати, там еще есть и ресурс с датой, до которой программа может работать, думаю, можно тоже подправить. И, возможно, полностью снести SIM LOCK. Не знаю, результатом доволен. Модем снова зарегистрировался в сети life:)

AT+CREG?
+CREG: 0,1

OK
AT^CVOICE?
^CVOICE:0,8000,16,20

OK

Версия прошивки - 11.030.01.07.388

The world is full of colors!

You can install Color Help application from Android Market.

I've always had problems identifying the colors. I kept avoiding color-related talks because I felt uncomfortable not being able to name the color, even though I saw that it was different. In 2007 I was finally diagnosed with a partial color blindness and my inability to identify the colors started making sense

The strange part of this is that given two cards of dark green and brown I had problems identifying which is which but when those cards were shown together I could name the colors. Then I found out that sometimes gray color is seen as purple and different light colors were really hard to identify.

But the world is full of colors and sometimes it is really required to tell the other person the color of an object, and it's better to be a correct one.

Fast-forward to today, now I have a mobile phone which can be programmed, the API is available and there are lots of examples. So I decided to write my own version of real-world color picker that will help me with my color problem

I wanted it to be a real-time application and it should have been as simple as possible. I also wanted it to be Open Source so that other people with the same issues as I could add/fix/propose something. And I wanted to test Android Market account.

You can get the Color Help application from Android Market for free and browse the source code in Launchpad

Planned features include better NV12 to RGB conversion that is using less CPU, fixes for occasional color detection hanging after returning to Color Help from another application and probably transformation of the stream to color-correct the image to finally be able to pass these crazy color blindness tests :)

I am now aware of the Color ID and Color Find applications, but being able to create something that scratches my own itch in a way I need it and make the source open is a nice exercise too!

Tomblog Note Publishing

Update (2011-09-16): At the moment the example web site is down. A number of users of Ubuntu One are unable to access CouchDB service due to poor CouchDB performance. The team is working on bringing additional hardware. If you are among the first 475000 users of Ubuntu One then this script will not work.

Screenshot

Installation notes

The project is quite new and it is currently a single-user installation. You can grab the code from lp:tomblog. It is easy to add Disqus-powered comments too, there is a JS code snippet in the template for that.

Please note that the code will allow displaying any note you have stored in your Ubuntu One database if the note UUID is known. The code will create links to the notes only if the target note is published, though.

In order to run tomblog on your server you will need to fill in your OAUTH_* credentials in settings-example.py and rename it to settings.py (I just took ones from my keyring), configure the templates dir, create the corresponding CouchDB views from data/views.js, install the following:

python-oauth
python-django
python-lxml

create the django wsgi application and configure apache server. As you see there is a lot of manual work currently but I plan to enable the service to be available for every Ubuntu One subscriber as a third party application utilizing Ubuntu One/CouchDB API.

Under the hood

When you sync your Tomboy notes with Ubuntu One server, the notes are stored in CouchDB on the server side.

This web application is basically a http client which fetches your notes tagged with some specific tag (in my case - Tomboy notebook called "Publish"), performs XML to HTML conversion and displays the published note.

It uses my OAuth token to retrieve the documents and my token can only be used with my CouchDB databases.

Live Example

The current trunk is running on notes.rtg.in.ua and looks to be pretty working. Drop me a comment if you find this idea useful too.

^* GNote is not currently syncing using Snowy protocol. There is a script to sync GNote notes to local CouchDB called gussie, you can grab it from lp:~rye/+junk/gussie.