[Android Software/Hacking General [Developers Only]] Fnorder:
For custom screen density, it appears that two changes were needed. Note that this will masquerade as HIGH (240) on everything, so it's not a good idea for tablets. These are diffs against the OP's .apk:

--- ../2/com/android/vending/model/DeviceConfiguration.smali 2011-10-31 20:51:53.000000000 +0000
+++ com/android/vending/model/DeviceConfiguration.smali 2011-11-03 20:43:13.000000000 +0000
@@ -1174,16 +1174,21 @@
.parameter "screenDensity"

.prologue
.line 302
iget-object v0, p0, Lcom/android/vending/model/DeviceConfiguration;->mProto:Lcom/google/common/io/pr

const/4 v1, 0x7

+# const/16 p1, 0x78 # let's lie about our ldpi density!
+# const/16 p1, 0xa0 # let's lie about our mdpi density!
+ const/16 p1, 0xf0 # let's lie about our hdpi density!
+# const/16 p1, 0x140 # let's lie about our xhdpi density!
+
invoke-virtual {v0, v1, p1}, Lcom/google/common/io/protocol/ProtoBuf;->setInt(II)V

.line 303
return-object p0
.end method

.method public setScreenHeight(I)Lcom/android/vending/model/DeviceConfiguration;
.registers 4

--- rel/com/android/vending/model/DeviceConfiguration$ScreenLayoutSize.smali 2011-10-31 20:51:53.000000000 +0000
+++ com/android/vending/model/DeviceConfiguration$ScreenLayoutSize.smali 2011-11-02 11:20:45.000000000 +0000

const/4 v5, 0x3

const/4 v4, 0x2

- const/4 v3, 0x2
+ const/4 v3, 0x2 # "SMALL" -> NORMAL

- const/4 v2, 0x0
+# const/4 v2, 0x0 # "UNDEFINED" -> google-deems-you-untrustworthy
+# const/4 v2, 0x1 # "UNDEFINED" -> SMALL
+# const/4 v2, 0x2 # "UNDEFINED" -> NORMAL
+ const/4 v2, 0x3 # "UNDEFINED" -> LARGE
+# const/4 v2, 0x4 # "UNDEFINED" -> EXTRA_LARGE

.line 170
new-instance v0, Lcom/android/vending/model/DeviceConfiguration$ScreenLayoutSize;

const-string v1, "UNDEFINED"

While testing I noticed /cache filling with apks of the latest market. This might be due to the fact that I didn't do the wipe-wipe-reboot-wipe-reboot thing and just did zip -u /system/app/Vending.apk;sync;reboot, but just incase the mod itself has an issue, and silent updates to crappy market 3 are bad, I neutered it:

diff -rU5 ../2/com/android/vending/InitializeMarketAction$MetadataAndDownloadsAction.smali com/android/vending/InitializeMarketAction$MetadataAndDownloadsAction.smali
--- ../2/com/android/vending/InitializeMarketAction$MetadataAndDownloadsAction.smali 2011-10-31 20:51:52.000000000 +0000
+++ com/android/vending/InitializeMarketAction$MetadataAndDownloadsAction.smali 2011-11-03 20:47:26.000000000 +0000
@@ -370,21 +370,26 @@

if-ge v10, v3, :cond_5e

.line 288
:cond_52
- invoke-virtual {v7}, Lcom/android/vending/model/GetMarketMetadataResponse;->getLatestClientUrl()Ljava/lang/String;

- move-result-object v2
+ const-string v10, "vendconf"
+ const-string v2, "rejecting self update"
+ invoke-static {v10, v2}, Landroid/util/Log;->d(Ljava/lang/String;Ljava/lang/String;)I
+
+# invoke-virtual {v7}, Lcom/android/vending/model/GetMarketMetadataResponse;->getLatestClientUrl()Ljava/lang/String;
+
+# move-result-object v2

.line 289
.local v2, latestClientUrl:Ljava/lang/String;
- new-instance v10, Lcom/android/vending/InitializeMarketAction$MetadataAndDownloadsAction$2;
+# new-instance v10, Lcom/android/vending/InitializeMarketAction$MetadataAndDownloadsAction$2;

- invoke-direct {v10, p0, v0, v2}, Lcom/android/vending/InitializeMarketAction$MetadataAndDownloadsAction$2;-><init>(Lcom/android/vending/InitializeMarketAction$MetadataAndDownloadsAction;Landroid/
content/Context;Ljava/lang/String;)V
+# invoke-direct {v10, p0, v0, v2}, Lcom/android/vending/InitializeMarketAction$MetadataAndDownloadsAction$2;-><init>(Lcom/android/vending/InitializeMarketAction$MetadataAndDownloadsAction;Landroid/
content/Context;Ljava/lang/String;)V

- invoke-virtual {v10}, Lcom/android/vending/InitializeMarketAction$MetadataAndDownloadsAction$2;->start()V
+# invoke-virtual {v10}, Lcom/android/vending/InitializeMarketAction$MetadataAndDownloadsAction$2;->start()V

.line 298
.end local v2 #latestClientUrl:Ljava/lang/String;
:cond_5e
invoke-virtual {v7}, Lcom/android/vending/model/GetMarketMetadataResponse;->hasBillingParameters()Z