krmin/mahjong_vision

Mahjong Vision Assistant

This project uses computer vision and machine learning to provide real-time discard suggestions for the game Mahjong Soul.

Features

• Tile Recognition: Identifies Mahjong tiles from the Mahjong Soul game window using a fine-tuned Vision Transformer model (pjura/mahjong_soul_vision).
• Game State Analysis: Parses the recognized tiles to understand the current game state (player's hand, melds, discard pools).
• Discard Suggestion: Employs a neural network (ImprovedNN), based on the architecture from the pjura/mahjong_ai repository, to predict the optimal discard based on the analyzed game state.
• Live Overlay: Captures the game window and overlays suggestions directly onto the screen, highlighting the recommended discard tile.

Project Structure

• live_feed.py: The main script to run the live assistant. It captures the screen, performs tile recognition, predicts discards, and displays the overlay.
• hf_vision_model.ipynb: Jupyter notebook detailing the training process for the Hugging Face Vision Transformer used for tile recognition.
• tools.py: Contains utility functions for data processing, model prediction, loss calculation, MLflow interaction, and tile representation translation used by live_feed.py. Many cross repo functions.
• model.safetensors: Saved weights for the discard prediction neural network (ImprovedNN).

Setup

1. Environment: Ensure you have Python installed along with necessary libraries. Key libraries include:

   • torch (with CUDA support if available)
   • transformers
   • datasets
   • evaluate
   • opencv-python (cv2)
   • Pillow (PIL)
   • pygetwindow
   • numpy
   • pyautogui
   • keyboard
   • safetensors
   • mlflow (Optional, used in tools.py, you can use whatever you like to serve the model)
   • scipy
   • matplotlib

   (A requirements.txt file would be beneficial here, but didn't made one at the time)

2. Models:

   • The tile recognition model (pjura/mahjong_soul_vision) will be downloaded automatically by the transformers library.
   • The discard prediction model (model.safetensors) should be present in the root directory.

Usage

1. Ensure the Mahjong Soul game window is open and titled "MahjongSoul".
2. Run the main script:

   python live_feed.py
   

3. The script will capture the game window, analyze the tiles, and highlight the suggested discard tile in the player's hand region. The color of the highlight indicates the model's confidence (Green=High, Red=Low).
4. Press 'q' to quit the application.
5. Auto-Click: When it is your turn (14 tiles in hand/melds) and a suggestion is highlighted, hold the Spacebar to automatically move the mouse and click the suggested tile. If Spacebar is not held, only the highlight will be shown.

Notes

• The script relies on specific window coordinates and aspect ratios which might need adjustment depending on screen resolution and game layout.
• The discard prediction model architecture (ImprovedNN) originates from the pjura/mahjong_ai repository. The included model.safetensors file is an example set of weights for this model, also from that repository, but potentially not the latest version. It was trained on the pjura/mahjong_board_states dataset, primarily using the tenhou_prediction_deepLearning_basic.ipynb notebook as detailed on the model card. You can add your own logic to load different weights or the latest version from the Hub.

This model is a fine-tuned version of google/vit-base-patch16-224-in21k on a local imagefolder dataset consisting of pictures of Mahjong tiles. It achieves the following results on the evaluation set:

• Loss: 0.0466
• Accuracy: 0.9967
• F1: 0.9966
• Recall: 0.9967

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 250

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1	Recall
3.5154	1.0	17	3.5109	0.0234	0.0154	0.0234
3.4741	2.0	34	3.4796	0.0769	0.0703	0.0769
3.3627	3.0	51	3.4305	0.1661	0.1266	0.1661
3.2456	4.0	68	3.3608	0.2230	0.1652	0.2230
3.1598	5.0	85	3.2658	0.2676	0.1989	0.2676
2.9972	6.0	102	3.1531	0.3467	0.2807	0.3467
2.7832	7.0	119	3.0176	0.4749	0.4135	0.4749
2.6689	8.0	136	2.8651	0.5507	0.4891	0.5507
2.3725	9.0	153	2.6983	0.6734	0.6192	0.6734
2.1117	10.0	170	2.5176	0.7570	0.7124	0.7570
1.9014	11.0	187	2.3488	0.8105	0.7771	0.8105
1.6784	12.0	204	2.1735	0.8618	0.8440	0.8618
1.4541	13.0	221	2.0088	0.9164	0.9092	0.9164
1.3576	14.0	238	1.8511	0.9487	0.9463	0.9487
1.2025	15.0	255	1.6971	0.9721	0.9718	0.9721
1.0567	16.0	272	1.5578	0.9844	0.9842	0.9844
0.898	17.0	289	1.4185	0.9889	0.9887	0.9889
0.7663	18.0	306	1.2978	0.9900	0.9899	0.9900
0.7498	19.0	323	1.1911	0.9911	0.9910	0.9911
0.6427	20.0	340	1.0966	0.9900	0.9899	0.9900
0.616	21.0	357	1.0003	0.9911	0.9910	0.9911
0.4962	22.0	374	0.9015	0.9900	0.9900	0.9900
0.4871	23.0	391	0.8413	0.9900	0.9899	0.9900
0.4257	24.0	408	0.7768	0.9911	0.9910	0.9911
0.3961	25.0	425	0.7042	0.9933	0.9933	0.9933
0.3513	26.0	442	0.6645	0.9922	0.9922	0.9922
0.3294	27.0	459	0.6179	0.9911	0.9911	0.9911
0.3021	28.0	476	0.5852	0.9900	0.9899	0.9900
0.2726	29.0	493	0.5444	0.9933	0.9933	0.9933
0.257	30.0	510	0.5177	0.9911	0.9910	0.9911
0.2382	31.0	527	0.4924	0.9900	0.9899	0.9900
0.2222	32.0	544	0.4582	0.9933	0.9933	0.9933
0.2059	33.0	561	0.4408	0.9922	0.9922	0.9922
0.1928	34.0	578	0.4222	0.9911	0.9910	0.9911
0.1864	35.0	595	0.3997	0.9922	0.9922	0.9922
0.176	36.0	612	0.3844	0.9922	0.9922	0.9922
0.1625	37.0	629	0.3693	0.9922	0.9922	0.9922
0.154	38.0	646	0.3539	0.9922	0.9921	0.9922
0.1524	39.0	663	0.3380	0.9933	0.9933	0.9933
0.1415	40.0	680	0.3256	0.9933	0.9933	0.9933
0.1362	41.0	697	0.3147	0.9922	0.9922	0.9922
0.1307	42.0	714	0.3023	0.9933	0.9933	0.9933
0.1263	43.0	731	0.2914	0.9944	0.9944	0.9944
0.1185	44.0	748	0.2811	0.9944	0.9944	0.9944
0.1143	45.0	765	0.2708	0.9944	0.9944	0.9944
0.109	46.0	782	0.2646	0.9933	0.9933	0.9933
0.1023	47.0	799	0.2564	0.9944	0.9944	0.9944
0.1	48.0	816	0.2472	0.9944	0.9944	0.9944
0.0969	49.0	833	0.2409	0.9944	0.9944	0.9944
0.0931	50.0	850	0.2336	0.9944	0.9944	0.9944
0.0926	51.0	867	0.2266	0.9944	0.9944	0.9944
0.0874	52.0	884	0.2217	0.9933	0.9933	0.9933
0.0837	53.0	901	0.2134	0.9944	0.9944	0.9944
0.0796	54.0	918	0.2099	0.9933	0.9933	0.9933
0.0759	55.0	935	0.2038	0.9944	0.9944	0.9944
0.0745	56.0	952	0.1987	0.9944	0.9944	0.9944
0.0745	57.0	969	0.1937	0.9944	0.9944	0.9944
0.0678	58.0	986	0.1883	0.9944	0.9944	0.9944
0.0666	59.0	1003	0.1841	0.9944	0.9944	0.9944
0.0642	60.0	1020	0.1805	0.9944	0.9944	0.9944
0.0608	61.0	1037	0.1756	0.9944	0.9944	0.9944
0.0615	62.0	1054	0.1724	0.9944	0.9944	0.9944
0.0582	63.0	1071	0.1689	0.9944	0.9944	0.9944
0.0574	64.0	1088	0.1650	0.9944	0.9944	0.9944
0.0558	65.0	1105	0.1612	0.9944	0.9944	0.9944
0.0551	66.0	1122	0.1581	0.9944	0.9944	0.9944
0.054	67.0	1139	0.1550	0.9944	0.9944	0.9944
0.0529	68.0	1156	0.1516	0.9944	0.9944	0.9944
0.0508	69.0	1173	0.1491	0.9944	0.9944	0.9944
0.0497	70.0	1190	0.1462	0.9944	0.9944	0.9944
0.0469	71.0	1207	0.1436	0.9944	0.9944	0.9944
0.0478	72.0	1224	0.1417	0.9933	0.9933	0.9933
0.0433	73.0	1241	0.1384	0.9944	0.9944	0.9944
0.0406	74.0	1258	0.1359	0.9944	0.9944	0.9944
0.0432	75.0	1275	0.1337	0.9955	0.9955	0.9955
0.0425	76.0	1292	0.1315	0.9944	0.9944	0.9944
0.0393	77.0	1309	0.1297	0.9944	0.9944	0.9944
0.0405	78.0	1326	0.1270	0.9944	0.9944	0.9944
0.0403	79.0	1343	0.1250	0.9955	0.9955	0.9955
0.037	80.0	1360	0.1233	0.9944	0.9944	0.9944
0.0377	81.0	1377	0.1213	0.9944	0.9944	0.9944
0.0336	82.0	1394	0.1195	0.9955	0.9955	0.9955
0.0366	83.0	1411	0.1174	0.9955	0.9955	0.9955
0.0361	84.0	1428	0.1156	0.9955	0.9955	0.9955
0.0351	85.0	1445	0.1140	0.9955	0.9955	0.9955
0.0333	86.0	1462	0.1126	0.9955	0.9955	0.9955
0.0343	87.0	1479	0.1109	0.9967	0.9966	0.9967
0.0316	88.0	1496	0.1096	0.9955	0.9955	0.9955
0.0319	89.0	1513	0.1077	0.9955	0.9955	0.9955
0.0297	90.0	1530	0.1062	0.9967	0.9966	0.9967
0.0285	91.0	1547	0.1050	0.9967	0.9966	0.9967
0.0288	92.0	1564	0.1037	0.9967	0.9966	0.9967
0.0283	93.0	1581	0.1026	0.9967	0.9966	0.9967
0.0282	94.0	1598	0.1011	0.9967	0.9966	0.9967
0.0281	95.0	1615	0.1001	0.9967	0.9966	0.9967
0.0283	96.0	1632	0.0986	0.9967	0.9966	0.9967
0.0274	97.0	1649	0.0976	0.9967	0.9966	0.9967
0.0261	98.0	1666	0.0965	0.9955	0.9955	0.9955
0.0249	99.0	1683	0.0955	0.9967	0.9966	0.9967
0.0252	100.0	1700	0.0941	0.9967	0.9966	0.9967
0.0258	101.0	1717	0.0930	0.9967	0.9966	0.9967
0.024	102.0	1734	0.0921	0.9967	0.9966	0.9967
0.0244	103.0	1751	0.0910	0.9967	0.9966	0.9967
0.0226	104.0	1768	0.0904	0.9967	0.9966	0.9967
0.0238	105.0	1785	0.0890	0.9967	0.9966	0.9967
0.0233	106.0	1802	0.0881	0.9967	0.9966	0.9967
0.0219	107.0	1819	0.0870	0.9967	0.9966	0.9967
0.0213	108.0	1836	0.0863	0.9967	0.9966	0.9967
0.0221	109.0	1853	0.0855	0.9967	0.9966	0.9967
0.0209	110.0	1870	0.0848	0.9967	0.9966	0.9967
0.0207	111.0	1887	0.0838	0.9967	0.9966	0.9967
0.0203	112.0	1904	0.0828	0.9967	0.9966	0.9967
0.0203	113.0	1921	0.0823	0.9967	0.9966	0.9967
0.0193	114.0	1938	0.0814	0.9967	0.9966	0.9967
0.0199	115.0	1955	0.0806	0.9967	0.9966	0.9967
0.0202	116.0	1972	0.0799	0.9967	0.9966	0.9967
0.0192	117.0	1989	0.0790	0.9967	0.9966	0.9967
0.0193	118.0	2006	0.0784	0.9967	0.9966	0.9967
0.0189	119.0	2023	0.0779	0.9967	0.9966	0.9967
0.0189	120.0	2040	0.0772	0.9967	0.9966	0.9967
0.0176	121.0	2057	0.0765	0.9967	0.9966	0.9967
0.0184	122.0	2074	0.0761	0.9967	0.9966	0.9967
0.0169	123.0	2091	0.0754	0.9967	0.9966	0.9967
0.0177	124.0	2108	0.0746	0.9967	0.9966	0.9967
0.0173	125.0	2125	0.0739	0.9967	0.9966	0.9967
0.0173	126.0	2142	0.0737	0.9967	0.9966	0.9967
0.016	127.0	2159	0.0729	0.9967	0.9966	0.9967
0.0167	128.0	2176	0.0724	0.9967	0.9966	0.9967
0.0164	129.0	2193	0.0714	0.9967	0.9966	0.9967
0.0158	130.0	2210	0.0711	0.9967	0.9966	0.9967
0.016	131.0	2227	0.0706	0.9967	0.9966	0.9967
0.0159	132.0	2244	0.0701	0.9967	0.9966	0.9967
0.0154	133.0	2261	0.0697	0.9967	0.9966	0.9967
0.0149	134.0	2278	0.0694	0.9967	0.9966	0.9967
0.0149	135.0	2295	0.0685	0.9967	0.9966	0.9967
0.0148	136.0	2312	0.0681	0.9967	0.9966	0.9967
0.0146	137.0	2329	0.0677	0.9967	0.9966	0.9967
0.0147	138.0	2346	0.0671	0.9967	0.9966	0.9967
0.0147	139.0	2363	0.0667	0.9967	0.9966	0.9967
0.0143	140.0	2380	0.0662	0.9967	0.9966	0.9967
0.0137	141.0	2397	0.0660	0.9967	0.9966	0.9967
0.0138	142.0	2414	0.0656	0.9967	0.9966	0.9967
0.0142	143.0	2431	0.0649	0.9967	0.9966	0.9967
0.0137	144.0	2448	0.0645	0.9967	0.9966	0.9967
0.0137	145.0	2465	0.0641	0.9967	0.9966	0.9967
0.0134	146.0	2482	0.0636	0.9967	0.9966	0.9967
0.014	147.0	2499	0.0632	0.9967	0.9966	0.9967
0.0132	148.0	2516	0.0632	0.9967	0.9966	0.9967
0.0135	149.0	2533	0.0627	0.9967	0.9966	0.9967
0.0128	150.0	2550	0.0624	0.9967	0.9966	0.9967
0.0123	151.0	2567	0.0619	0.9967	0.9966	0.9967
0.0124	152.0	2584	0.0615	0.9967	0.9966	0.9967
0.0127	153.0	2601	0.0609	0.9967	0.9966	0.9967
0.0127	154.0	2618	0.0607	0.9967	0.9966	0.9967
0.0124	155.0	2635	0.0607	0.9967	0.9966	0.9967
0.0121	156.0	2652	0.0601	0.9967	0.9966	0.9967
0.0118	157.0	2669	0.0599	0.9967	0.9966	0.9967
0.0123	158.0	2686	0.0596	0.9967	0.9966	0.9967
0.0118	159.0	2703	0.0590	0.9967	0.9966	0.9967
0.0116	160.0	2720	0.0589	0.9967	0.9966	0.9967
0.0112	161.0	2737	0.0586	0.9967	0.9966	0.9967
0.0113	162.0	2754	0.0582	0.9967	0.9966	0.9967
0.0116	163.0	2771	0.0579	0.9967	0.9966	0.9967
0.011	164.0	2788	0.0576	0.9967	0.9966	0.9967
0.0114	165.0	2805	0.0575	0.9967	0.9966	0.9967
0.0109	166.0	2822	0.0572	0.9967	0.9966	0.9967
0.0102	167.0	2839	0.0569	0.9967	0.9966	0.9967
0.0106	168.0	2856	0.0568	0.9967	0.9966	0.9967
0.0103	169.0	2873	0.0564	0.9967	0.9966	0.9967
0.0105	170.0	2890	0.0561	0.9967	0.9966	0.9967
0.0106	171.0	2907	0.0560	0.9967	0.9966	0.9967
0.01	172.0	2924	0.0556	0.9967	0.9966	0.9967
0.0098	173.0	2941	0.0554	0.9967	0.9966	0.9967
0.0098	174.0	2958	0.0550	0.9967	0.9966	0.9967
0.0107	175.0	2975	0.0549	0.9967	0.9966	0.9967
0.0103	176.0	2992	0.0546	0.9967	0.9966	0.9967
0.0104	177.0	3009	0.0544	0.9967	0.9966	0.9967
0.0096	178.0	3026	0.0542	0.9967	0.9966	0.9967
0.0102	179.0	3043	0.0540	0.9967	0.9966	0.9967
0.0097	180.0	3060	0.0538	0.9967	0.9966	0.9967
0.0096	181.0	3077	0.0535	0.9967	0.9966	0.9967
0.0093	182.0	3094	0.0536	0.9967	0.9966	0.9967
0.0097	183.0	3111	0.0531	0.9967	0.9966	0.9967
0.0093	184.0	3128	0.0529	0.9967	0.9966	0.9967
0.0097	185.0	3145	0.0526	0.9967	0.9966	0.9967
0.0094	186.0	3162	0.0527	0.9967	0.9966	0.9967
0.0095	187.0	3179	0.0524	0.9967	0.9966	0.9967
0.0093	188.0	3196	0.0522	0.9967	0.9966	0.9967
0.0089	189.0	3213	0.0520	0.9967	0.9966	0.9967
0.0091	190.0	3230	0.0520	0.9967	0.9966	0.9967
0.0091	191.0	3247	0.0516	0.9967	0.9966	0.9967
0.009	192.0	3264	0.0515	0.9967	0.9966	0.9967
0.009	193.0	3281	0.0514	0.9967	0.9966	0.9967
0.0091	194.0	3298	0.0512	0.9967	0.9966	0.9967
0.009	195.0	3315	0.0509	0.9967	0.9966	0.9967
0.0087	196.0	3332	0.0510	0.9967	0.9966	0.9967
0.009	197.0	3349	0.0507	0.9967	0.9966	0.9967
0.0087	198.0	3366	0.0506	0.9967	0.9966	0.9967
0.0084	199.0	3383	0.0505	0.9967	0.9966	0.9967
0.009	200.0	3400	0.0503	0.9967	0.9966	0.9967
0.0087	201.0	3417	0.0501	0.9967	0.9966	0.9967
0.0088	202.0	3434	0.0500	0.9967	0.9966	0.9967
0.0086	203.0	3451	0.0500	0.9967	0.9966	0.9967
0.0085	204.0	3468	0.0497	0.9967	0.9966	0.9967
0.009	205.0	3485	0.0496	0.9967	0.9966	0.9967
0.0082	206.0	3502	0.0495	0.9967	0.9966	0.9967
0.008	207.0	3519	0.0494	0.9967	0.9966	0.9967
0.0082	208.0	3536	0.0493	0.9967	0.9966	0.9967
0.0078	209.0	3553	0.0491	0.9967	0.9966	0.9967
0.0082	210.0	3570	0.0490	0.9967	0.9966	0.9967
0.0082	211.0	3587	0.0489	0.9967	0.9966	0.9967
0.0085	212.0	3604	0.0488	0.9967	0.9966	0.9967
0.0087	213.0	3621	0.0487	0.9967	0.9966	0.9967
0.0079	214.0	3638	0.0485	0.9967	0.9966	0.9967
0.0078	215.0	3655	0.0484	0.9967	0.9966	0.9967
0.0078	216.0	3672	0.0484	0.9967	0.9966	0.9967
0.0082	217.0	3689	0.0483	0.9967	0.9966	0.9967
0.0085	218.0	3706	0.0482	0.9967	0.9966	0.9967
0.0079	219.0	3723	0.0480	0.9967	0.9966	0.9967
0.0079	220.0	3740	0.0480	0.9967	0.9966	0.9967
0.0076	221.0	3757	0.0479	0.9967	0.9966	0.9967
0.008	222.0	3774	0.0478	0.9967	0.9966	0.9967
0.0078	223.0	3791	0.0477	0.9967	0.9966	0.9967
0.0078	224.0	3808	0.0476	0.9967	0.9966	0.9967
0.0078	225.0	3825	0.0476	0.9967	0.9966	0.9967
0.0077	226.0	3842	0.0475	0.9967	0.9966	0.9967
0.0075	227.0	3859	0.0475	0.9967	0.9966	0.9967
0.0075	228.0	3876	0.0474	0.9967	0.9966	0.9967
0.0076	229.0	3893	0.0473	0.9967	0.9966	0.9967
0.0077	230.0	3910	0.0472	0.9967	0.9966	0.9967
0.0076	231.0	3927	0.0472	0.9967	0.9966	0.9967
0.0074	232.0	3944	0.0471	0.9967	0.9966	0.9967
0.0076	233.0	3961	0.0471	0.9967	0.9966	0.9967
0.0074	234.0	3978	0.0470	0.9967	0.9966	0.9967
0.0077	235.0	3995	0.0470	0.9967	0.9966	0.9967
0.0074	236.0	4012	0.0469	0.9967	0.9966	0.9967
0.0075	237.0	4029	0.0469	0.9967	0.9966	0.9967
0.0072	238.0	4046	0.0469	0.9967	0.9966	0.9967
0.0075	239.0	4063	0.0468	0.9967	0.9966	0.9967
0.0078	240.0	4080	0.0468	0.9967	0.9966	0.9967
0.0075	241.0	4097	0.0468	0.9967	0.9966	0.9967
0.0073	242.0	4114	0.0468	0.9967	0.9966	0.9967
0.0073	243.0	4131	0.0467	0.9967	0.9966	0.9967
0.0068	244.0	4148	0.0467	0.9967	0.9966	0.9967
0.0072	245.0	4165	0.0467	0.9967	0.9966	0.9967
0.0073	246.0	4182	0.0467	0.9967	0.9966	0.9967
0.0077	247.0	4199	0.0467	0.9967	0.9966	0.9967
0.0074	248.0	4216	0.0466	0.9967	0.9966	0.9967
0.0073	249.0	4233	0.0466	0.9967	0.9966	0.9967
0.0074	250.0	4250	0.0466	0.9967	0.9966	0.9967

Framework versions

• Transformers 4.34.1
• Pytorch 2.1.0+cu121
• Datasets 2.14.6
• Tokenizers 0.14.1