When I initially reviewed the Compute Module 4 IO Board, I briefly mentioned there’s a 4-pin fan connector. It’s connected to the Pi’s I2C bus using a little PWM chip, the EMC2301.

But wait… what’s I2C, what’s PWM, and what’s so special about a 4-pin fan connector? I’m glad you asked—this post will answer that and show you how you can control a fan connected to the IO Board, like the quiet Noctua NF-P12 pictured above with my IO Board.

If you plug a fan like that into the CM4 IO Board, it will start running full blast, 24×7. If you need that much cooling, that’s great, but a lot of times, I don’t mind my Pi’s CPU getting warmer if it means I can run the fan silent most of the time.

So what are my options? First of all, I could just buy an inline PWM controller, like a Noctua NA-FC1. It lets me turn up and down the fan speed with a little dial. But it doesn’t know the temperature of my Pi, so it can’t increase airflow for higher temperatures or turn off the fan when it’s under a certain temperature.

The better option is to use the built-in PWM fan controller on the IO Board (pictured above). And to do that, we’re going to need to use the Raspberry Pi’s I2C bus!

What is I2C?

I2C—or more correctly, I²C—stands for “Inter-Integrated Circuit” and is a two-wire serial communication interface used by many electronic devices for control and communications.

I’m not going to cover it in detail here, but if you get into any more advanced electronics projects with Arduino, Raspberry Pi, or other microcontrollers or PCs, you’ll probably encounter it. To learn the basics of the protocol, I recommend Analog Device’s I²C Primer.

Controlling the fan over I²C

You have to edit your /boot/config.txt file to enable the i2c_vc bus, which is bus #1. The Pi Device Tree Documentation actually recommends against touching i2c_vc unless you need to, because you could mess up CSI camera or DSI display functionality.

Make sure the following lines exist and are uncommented in /boot/config.txt and reboot the Pi:

# Enable I2C.
dtparam=i2c_arm=on
# Enable I2C bus 1.
dtparam=i2c_vc=on

Note: If you just enable I²C under the ‘Interfaces’ option of raspi-config, it will only enable i2c_arm. To see the fan controller, you need to enable i2c_vc as well.

Make sure the i2c-tools package is installed on your system; if it is, the following commands should work straightaway. If not, you will need to install the package with sudo apt-get install -y i2c-tools.

Now, check if you can see the fan controller chip on the bus, using i2cdetect -y 10:

$ i2cdetect -y 10
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- 0c -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 2f
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- 51 -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: -- -- -- -- -- -- -- --

The fan is the 2f device. Test if you can turn off the fan using:

$ i2cset -y 10 0x2f 0x30 0x00

The fan should now be off. And to turn it back on:

$ i2cset -y 10 0x2f 0x30 0xff

To get the value of the fan setting, you can use:

$ i2cget -y 10 0x2f 0x30
0xff

What about setting the fan to a value between 0% (off) and 100% (full on) though? The value is hexadecimal, so 0xFF stands for 255, while 0x00 is 0. Using a high-ish number should be safe, right, to set the fan to a lower speed? Well, let’s try it out.

Load more 4501
Load more 4502
Load more 4503
Load more 4504
Load more 4505
Load more 4506
Load more 4507
Load more 4508
Load more 4509
Load more 4510
Load more 4511
Load more 4512
Load more 4513
Load more 4514
Load more 4515
Load more 4516
Load more 4517
Load more 4518
Load more 4519
Load more 4520
Load more 4521
Load more 4522
Load more 4523
Load more 4524
Load more 4525
Load more 4526
Load more 4527
Load more 4528
Load more 4529
Load more 4530
Load more 4531
Load more 4532
Load more 4533
Load more 4534
Load more 4535
Load more 4536
Load more 4537
Load more 4538
Load more 4539
Load more 4540
Load more 4541
Load more 4542
Load more 4543
Load more 4544
Load more 4545
Load more 4546
Load more 4547
Load more 4548
Load more 4549
Load more 4550
Load more 4551
Load more 4552
Load more 4553
Load more 4554
Load more 4555
Load more 4556
Load more 4557
Load more 4558
Load more 4559
Load more 4560
Load more 4561
Load more 4562
Load more 4563
Load more 4564
Load more 4565
Load more 4566
Load more 4567
Load more 4568
Load more 4569
Load more 4570
Load more 4571
Load more 4572
Load more 4573
Load more 4574
Load more 4575
Load more 4576
Load more 4577
Load more 4578
Load more 4579
Load more 4580
Load more 4581
Load more 4582
Load more 4583
Load more 4584
Load more 4585
Load more 4586
Load more 4587
Load more 4588
Load more 4589
Load more 4590
Load more 4591
Load more 4592
Load more 4593
Load more 4594
Load more 4595
Load more 4596
Load more 4597
Load more 4598
Load more 4599
Load more 4600
Load more 4601
Load more 4602
Load more 4603
Load more 4604
Load more 4605
Load more 4606
Load more 4607
Load more 4608
Load more 4609
Load more 4610
Load more 4611
Load more 4612
Load more 4613
Load more 4614
Load more 4615
Load more 4616
Load more 4617
Load more 4618
Load more 4619
Load more 4620
Load more 4621
Load more 4622
Load more 4623
Load more 4624
Load more 4625
Load more 4626
Load more 4627
Load more 4628
Load more 4629
Load more 4630
Load more 4631
Load more 4632
Load more 4633
Load more 4634
Load more 4635
Load more 4636
Load more 4637
Load more 4638
Load more 4639
Load more 4640
Load more 4641
Load more 4642
Load more 4643
Load more 4644
Load more 4645
Load more 4646
Load more 4647
Load more 4648
Load more 4649
Load more 4650
Load more 4651
Load more 4652
Load more 4653
Load more 4654
Load more 4655
Load more 4656
Load more 4657
Load more 4658
Load more 4659
Load more 4660
Load more 4661
Load more 4662
Load more 4663
Load more 4664
Load more 4665
Load more 4666
Load more 4667
Load more 4668
Load more 4669
Load more 4670
Load more 4671
Load more 4672
Load more 4673
Load more 4674
Load more 4675
Load more 4676
Load more 4677
Load more 4678
Load more 4679
Load more 4680
Load more 4681
Load more 4682
Load more 4683
Load more 4684
Load more 4685
Load more 4686
Load more 4687
Load more 4688
Load more 4689
Load more 4690
Load more 4691
Load more 4692
Load more 4693
Load more 4694
Load more 4695
Load more 4696
Load more 4697
Load more 4698
Load more 4699
Load more 4700
Load more 4701
Load more 4702
Load more 4703
Load more 4704
Load more 4705
Load more 4706
Load more 4707
Load more 4708
Load more 4709
Load more 4710
Load more 4711
Load more 4712
Load more 4713
Load more 4714
Load more 4715
Load more 4716
Load more 4717
Load more 4718
Load more 4719
Load more 4720
Load more 4721
Load more 4722
Load more 4723
Load more 4724
Load more 4725
Load more 4726
Load more 4727
Load more 4728
Load more 4729
Load more 4730
Load more 4731
Load more 4732
Load more 4733
Load more 4734
Load more 4735
Load more 4736
Load more 4737
Load more 4738
Load more 4739
Load more 4740
Load more 4741
Load more 4742
Load more 4743
Load more 4744
Load more 4745
Load more 4746
Load more 4747
Load more 4748
Load more 4749
Load more 4750
Load more 4751
Load more 4752
Load more 4753
Load more 4754
Load more 4755
Load more 4756
Load more 4757
Load more 4758
Load more 4759
Load more 4760
Load more 4761
Load more 4762
Load more 4763
Load more 4764
Load more 4765
Load more 4766
Load more 4767
Load more 4768
Load more 4769
Load more 4770
Load more 4771
Load more 4772
Load more 4773
Load more 4774
Load more 4775
Load more 4776
Load more 4777
Load more 4778
Load more 4779
Load more 4780
Load more 4781
Load more 4782
Load more 4783
Load more 4784
Load more 4785
Load more 4786
Load more 4787
Load more 4788
Load more 4789
Load more 4790
Load more 4791
Load more 4792
Load more 4793
Load more 4794
Load more 4795
Load more 4796
Load more 4797
Load more 4798
Load more 4799
Load more 4800
Load more 4801
Load more 4802
Load more 4803
Load more 4804
Load more 4805
Load more 4806
Load more 4807
Load more 4808
Load more 4809
Load more 4810
Load more 4811
Load more 4812
Load more 4813
Load more 4814
Load more 4815
Load more 4816
Load more 4817
Load more 4818
Load more 4819
Load more 4820
Load more 4821
Load more 4822
Load more 4823
Load more 4824
Load more 4825
Load more 4826
Load more 4827
Load more 4828
Load more 4829
Load more 4830
Load more 4831
Load more 4832
Load more 4833
Load more 4834
Load more 4835
Load more 4836
Load more 4837
Load more 4838
Load more 4839
Load more 4840
Load more 4841
Load more 4842
Load more 4843
Load more 4844
Load more 4845
Load more 4846
Load more 4847
Load more 4848
Load more 4849
Load more 4850
Load more 4851
Load more 4852
Load more 4853
Load more 4854
Load more 4855
Load more 4856
Load more 4857
Load more 4858
Load more 4859
Load more 4860
Load more 4861
Load more 4862
Load more 4863
Load more 4864
Load more 4865
Load more 4866
Load more 4867
Load more 4868
Load more 4869
Load more 4870

Since the Compute Module 4 came along last year, there have been a few projects that use it that make me do a double-take: They did what with a Pi?

Alftel’s Seaberry is a carrier board for the CM4 in the Mini ITX form factor that adds on eleven PCI Express slots:

• 1 x16 slot (with x1 lane) in the standard ITX location
• 1 x1 slot on board edge
• 4 mini PCIe slots
• 4 M.2 E-key slots (with dual PCIe lines so you can run specialty cards like dual-TPU accelerators)
• 1 M.2 M-key slot for NVMe SSD

The rear of the board also has a decent mix of built-in IO, including a Cisco-style serial console cable you can use with an easy-to-find USB adapter so you can attach to a Raspberry Pi’s Serial Console (UART) for debugging.

I’ve been using the board for a couple weeks, even installing it inside my smallest ITX PC case, a Goodisory MX01. This is the first Pi board I’ve used that fits in a standard PC form factor, marking the first time I’ve been able to build a ‘custom PC’ with a Pi, using standardized components and not requiring extra time strapping together a makeshift or 3D-printed case:

And the Seaberry needs all that room, to jam in all the ports! I tested an Intel i350 dual-gigabit LAN card, a Dual-TPU Coral M.2 Accelerator, a KIOXIA XG6 M.2 NVMe SSD, a Compex WLE200NX 802.11n WiFi card, an Intel AX210 WiFi 6E card, a dual SATA controller, and more—and everything worked (more or less…).

Many cards have built-in drivers in Pi OS—the NVMe drive and the SATA SSD I plugged in worked right away, and required no custom drivers or kernel rebuilds. (For power, the board also includes a four-pin berg header to power SATA drives and other peripherals).

Cards like Intel’s AX210 required the installation of custom firmware and Intel’s driver, but worked great afterwards (giving me over 1 Gbps wireless on my home’s WiFi 6 network!).

Google’s Coral TPU cards still don’t work, unfortunately—the PCI Express implementation on the BCM2711 seems to not handle some of the memory access patterns it needs—but that brings me to another feature of the Seaberry (and many other CM4 boards): it is also supposedly compatible with Radxa’s CM3 and Pine64’s SOQuartz, two ‘pin compatible’ replacements for the CM4.

I have a CM3 and am awaiting shipment of the SOQuartz, so I’ll test them out later this year.

The other type of card I’m keen to get working with ARM64 SBCs is any kind of graphics card—mostly for the fun of it. I’ve struggled mightily to get AMD’s radeon and amdgpu drivers working, and Nvidia’s black box driver also has the same problem with the CM4—random hard lockups and kernel panics.

Anyways, the Seaberry is the perfect platform for my testing—and I even installed Alftel’s 12-card M.2 carrier board in the thing, so I could have 20 PCI Express devices wired into the Pi at once.

Doing so also uncovered one limitation with the CM4 (besides the anemic x1 Gen 2.0 lane): if I plugged in more than three NVMe drives at once, it seemed the nvme driver would have some strange kernel panics either at boot or shutdown. Sometimes it would boot fine with four NVMe drives, but never with five or more.

Load more 4001
Load more 4002
Load more 4003
Load more 4004
Load more 4005
Load more 4006
Load more 4007
Load more 4008
Load more 4009
Load more 4010
Load more 4011
Load more 4012
Load more 4013
Load more 4014
Load more 4015
Load more 4016
Load more 4017
Load more 4018
Load more 4019
Load more 4020
Load more 4021
Load more 4022
Load more 4023
Load more 4024
Load more 4025
Load more 4026
Load more 4027
Load more 4028
Load more 4029
Load more 4030
Load more 4031
Load more 4032
Load more 4033
Load more 4034
Load more 4035
Load more 4036
Load more 4037
Load more 4038
Load more 4039
Load more 4040
Load more 4041
Load more 4042
Load more 4043
Load more 4044
Load more 4045
Load more 4046
Load more 4047
Load more 4048
Load more 4049
Load more 4050
Load more 4051
Load more 4052
Load more 4053
Load more 4054
Load more 4055
Load more 4056
Load more 4057
Load more 4058
Load more 4059
Load more 4060
Load more 4061
Load more 4062
Load more 4063
Load more 4064
Load more 4065
Load more 4066
Load more 4067
Load more 4068
Load more 4069
Load more 4070
Load more 4071
Load more 4072
Load more 4073
Load more 4074
Load more 4075
Load more 4076
Load more 4077
Load more 4078
Load more 4079
Load more 4080
Load more 4081
Load more 4082
Load more 4083
Load more 4084
Load more 4085
Load more 4086
Load more 4087
Load more 4088
Load more 4089
Load more 4090
Load more 4091
Load more 4092
Load more 4093
Load more 4094
Load more 4095
Load more 4096
Load more 4097
Load more 4098
Load more 4099
Load more 4100
Load more 4101
Load more 4102
Load more 4103
Load more 4104
Load more 4105
Load more 4106
Load more 4107
Load more 4108
Load more 4109
Load more 4110
Load more 4111
Load more 4112
Load more 4113
Load more 4114
Load more 4115
Load more 4116
Load more 4117
Load more 4118
Load more 4119
Load more 4120
Load more 4121
Load more 4122
Load more 4123
Load more 4124
Load more 4125
Load more 4126
Load more 4127
Load more 4128
Load more 4129
Load more 4130
Load more 4131
Load more 4132
Load more 4133
Load more 4134
Load more 4135
Load more 4136
Load more 4137
Load more 4138
Load more 4139
Load more 4140
Load more 4141
Load more 4142
Load more 4143
Load more 4144
Load more 4145
Load more 4146
Load more 4147
Load more 4148
Load more 4149
Load more 4150
Load more 4151
Load more 4152
Load more 4153
Load more 4154
Load more 4155
Load more 4156
Load more 4157
Load more 4158
Load more 4159
Load more 4160
Load more 4161
Load more 4162
Load more 4163
Load more 4164
Load more 4165
Load more 4166
Load more 4167
Load more 4168
Load more 4169
Load more 4170
Load more 4171
Load more 4172
Load more 4173
Load more 4174
Load more 4175
Load more 4176
Load more 4177
Load more 4178
Load more 4179
Load more 4180
Load more 4181
Load more 4182
Load more 4183
Load more 4184
Load more 4185
Load more 4186
Load more 4187
Load more 4188
Load more 4189
Load more 4190
Load more 4191
Load more 4192
Load more 4193
Load more 4194
Load more 4195
Load more 4196
Load more 4197
Load more 4198
Load more 4199
Load more 4200
Load more 4201
Load more 4202
Load more 4203
Load more 4204
Load more 4205
Load more 4206
Load more 4207
Load more 4208
Load more 4209
Load more 4210
Load more 4211
Load more 4212
Load more 4213
Load more 4214
Load more 4215
Load more 4216
Load more 4217
Load more 4218
Load more 4219
Load more 4220
Load more 4221
Load more 4222
Load more 4223
Load more 4224
Load more 4225
Load more 4226
Load more 4227
Load more 4228
Load more 4229
Load more 4230
Load more 4231
Load more 4232
Load more 4233
Load more 4234
Load more 4235
Load more 4236
Load more 4237
Load more 4238
Load more 4239
Load more 4240
Load more 4241
Load more 4242
Load more 4243
Load more 4244
Load more 4245
Load more 4246
Load more 4247
Load more 4248
Load more 4249
Load more 4250
Load more 4251
Load more 4252
Load more 4253
Load more 4254
Load more 4255
Load more 4256
Load more 4257
Load more 4258
Load more 4259
Load more 4260
Load more 4261
Load more 4262
Load more 4263
Load more 4264
Load more 4265
Load more 4266
Load more 4267
Load more 4268
Load more 4269
Load more 4270
Load more 4271
Load more 4272
Load more 4273
Load more 4274
Load more 4275
Load more 4276
Load more 4277
Load more 4278
Load more 4279
Load more 4280
Load more 4281
Load more 4282
Load more 4283
Load more 4284
Load more 4285
Load more 4286
Load more 4287
Load more 4288
Load more 4289
Load more 4290
Load more 4291
Load more 4292
Load more 4293
Load more 4294
Load more 4295
Load more 4296
Load more 4297
Load more 4298
Load more 4299
Load more 4300
Load more 4301
Load more 4302
Load more 4303
Load more 4304
Load more 4305
Load more 4306
Load more 4307
Load more 4308
Load more 4309
Load more 4310
Load more 4311
Load more 4312
Load more 4313
Load more 4314
Load more 4315
Load more 4316
Load more 4317
Load more 4318
Load more 4319
Load more 4320
Load more 4321
Load more 4322
Load more 4323
Load more 4324
Load more 4325
Load more 4326
Load more 4327
Load more 4328
Load more 4329
Load more 4330
Load more 4331
Load more 4332
Load more 4333
Load more 4334
Load more 4335
Load more 4336
Load more 4337
Load more 4338
Load more 4339
Load more 4340
Load more 4341
Load more 4342
Load more 4343
Load more 4344
Load more 4345
Load more 4346
Load more 4347
Load more 4348
Load more 4349
Load more 4350
Load more 4351
Load more 4352
Load more 4353
Load more 4354
Load more 4355
Load more 4356
Load more 4357
Load more 4358
Load more 4359
Load more 4360
Load more 4361
Load more 4362
Load more 4363
Load more 4364
Load more 4365
Load more 4366
Load more 4367
Load more 4368
Load more 4369
Load more 4370
Load more 4371
Load more 4372
Load more 4373
Load more 4374
Load more 4375
Load more 4376
Load more 4377
Load more 4378
Load more 4379
Load more 4380
Load more 4381
Load more 4382
Load more 4383
Load more 4384
Load more 4385
Load more 4386
Load more 4387
Load more 4388
Load more 4389
Load more 4390
Load more 4391
Load more 4392
Load more 4393
Load more 4394
Load more 4395
Load more 4396
Load more 4397
Load more 4398
Load more 4399
Load more 4400
Load more 4401
Load more 4402
Load more 4403
Load more 4404
Load more 4405
Load more 4406
Load more 4407
Load more 4408
Load more 4409
Load more 4410
Load more 4411
Load more 4412
Load more 4413
Load more 4414
Load more 4415
Load more 4416
Load more 4417
Load more 4418
Load more 4419
Load more 4420
Load more 4421
Load more 4422
Load more 4423
Load more 4424
Load more 4425
Load more 4426
Load more 4427
Load more 4428
Load more 4429
Load more 4430
Load more 4431
Load more 4432
Load more 4433
Load more 4434
Load more 4435
Load more 4436
Load more 4437
Load more 4438
Load more 4439
Load more 4440
Load more 4441
Load more 4442
Load more 4443
Load more 4444
Load more 4445
Load more 4446
Load more 4447
Load more 4448
Load more 4449
Load more 4450
Load more 4451
Load more 4452
Load more 4453
Load more 4454
Load more 4455
Load more 4456
Load more 4457
Load more 4458
Load more 4459
Load more 4460
Load more 4461
Load more 4462
Load more 4463
Load more 4464
Load more 4465
Load more 4466
Load more 4467
Load more 4468
Load more 4469
Load more 4470
Load more 4471
Load more 4472
Load more 4473
Load more 4474
Load more 4475
Load more 4476
Load more 4477
Load more 4478
Load more 4479
Load more 4480
Load more 4481
Load more 4482
Load more 4483
Load more 4484
Load more 4485
Load more 4486
Load more 4487
Load more 4488
Load more 4489
Load more 4490
Load more 4491
Load more 4492
Load more 4493
Load more 4494
Load more 4495
Load more 4496
Load more 4497
Load more 4498
Load more 4499
Load more 4500

Last year I spent a bit of time building a Kubernetes cluster with the original Turing Pi. It was fun, and interesting, but ultimately the performance of the Compute Module 3+ it was designed around led me to running my homelab off some newer Pi 4 model B computers, which are at least twice as fast for almost everything I run on them.

So this year, I was excited when the folks at Turing Pi sent me a Turing Pi 2 to test drive. And the board arrived just in time for Patrick Kennedy from ServeTheHome to challenge me to a cluster build-off at Supercomputing ’21! Check out his ARM cluster build here.

The Turing Pi 2 is a mini ITX motherboard capable of holding up to four Raspberry Pi Compute Module 4s or NVIDIA Jetson Nanos, and it integrates a board management backplane, power management, and gigabit Ethernet switch, alongside various PCI Express breakouts, so you can build a 4-node SBC cluster.

And it performs—as I hoped—much better than the older version. Not only do the CM4, Jetson Nano, and even pin-compatible replacements like Radxa’s CM3 and Pine64’s SOQuartz all have full gigabit Ethernet, their CPUs are noticeably faster than the CM3+ they replace.

The board I have is a prototype, and as such is still running a very early version of the firmware—RTC PWM fan control aren’t even implemented yet! But the folks at Turing Pi seem set on having all the features ironed out for a January 2022 launch. The board will cost about $200, with CM4 adapter cards (which connect a CM4 to the vertical SO-DIMM slot) adding on $10 each.

PCIe Expansion support

Since each CM4 has one PCI Express Gen 2 lane available, the board exposes each one in a different way.

For node 1 (at the top of the board), there’s a mini PCIe slot with a SIM tray underneath—useful for things like 4G or 5G modems. For node 2, there’s another mini PCIe slot (with no SIM tray).

Node 3 is connected to an ASMedia 2-port SATA controller, so you can plug in up to two SATA III drives directly into the Turing Pi 2, and they’ll be controlled by the computer in slot 3.

Node 4 is connected to a VL805 USB 3.0 controller, and that exposes a USB 3 front panel header and two USB 3.0 ports on the rear to the computer in slot 4.

Blinkenlights

Besides performance, one thing I didn’t like about the original board was how few status LEDs it had. This new board is much better in that regard, with status LEDs for almost every important feature.

Both of the Ethernet ports on the back have functional link and activity lights. Each slot also has appropriately-colored link and activity lights. Each slot also has a power indicator LED, plus there’s an overall board power LED, and an LED to indicate the MCU is ‘on’ (helpful when deciding to safely shut down the cluster).

And finally, each of the CM4 adapter cards has a green power and activity LED on the back, so you can visually confirm they’re powered on and doing something.

All in all, a good show of blinkenlights if I’ve ever seen one.

Load more 3501
Load more 3502
Load more 3503
Load more 3504
Load more 3505
Load more 3506
Load more 3507
Load more 3508
Load more 3509
Load more 3510
Load more 3511
Load more 3512
Load more 3513
Load more 3514
Load more 3515
Load more 3516
Load more 3517
Load more 3518
Load more 3519
Load more 3520
Load more 3521
Load more 3522
Load more 3523
Load more 3524
Load more 3525
Load more 3526
Load more 3527
Load more 3528
Load more 3529
Load more 3530
Load more 3531
Load more 3532
Load more 3533
Load more 3534
Load more 3535
Load more 3536
Load more 3537
Load more 3538
Load more 3539
Load more 3540
Load more 3541
Load more 3542
Load more 3543
Load more 3544
Load more 3545
Load more 3546
Load more 3547
Load more 3548
Load more 3549
Load more 3550
Load more 3551
Load more 3552
Load more 3553
Load more 3554
Load more 3555
Load more 3556
Load more 3557
Load more 3558
Load more 3559
Load more 3560
Load more 3561
Load more 3562
Load more 3563
Load more 3564
Load more 3565
Load more 3566
Load more 3567
Load more 3568
Load more 3569
Load more 3570
Load more 3571
Load more 3572
Load more 3573
Load more 3574
Load more 3575
Load more 3576
Load more 3577
Load more 3578
Load more 3579
Load more 3580
Load more 3581
Load more 3582
Load more 3583
Load more 3584
Load more 3585
Load more 3586
Load more 3587
Load more 3588
Load more 3589
Load more 3590
Load more 3591
Load more 3592
Load more 3593
Load more 3594
Load more 3595
Load more 3596
Load more 3597
Load more 3598
Load more 3599
Load more 3600
Load more 3601
Load more 3602
Load more 3603
Load more 3604
Load more 3605
Load more 3606
Load more 3607
Load more 3608
Load more 3609
Load more 3610
Load more 3611
Load more 3612
Load more 3613
Load more 3614
Load more 3615
Load more 3616
Load more 3617
Load more 3618
Load more 3619
Load more 3620
Load more 3621
Load more 3622
Load more 3623
Load more 3624
Load more 3625
Load more 3626
Load more 3627
Load more 3628
Load more 3629
Load more 3630
Load more 3631
Load more 3632
Load more 3633
Load more 3634
Load more 3635
Load more 3636
Load more 3637
Load more 3638
Load more 3639
Load more 3640
Load more 3641
Load more 3642
Load more 3643
Load more 3644
Load more 3645
Load more 3646
Load more 3647
Load more 3648
Load more 3649
Load more 3650
Load more 3651
Load more 3652
Load more 3653
Load more 3654
Load more 3655
Load more 3656
Load more 3657
Load more 3658
Load more 3659
Load more 3660
Load more 3661
Load more 3662
Load more 3663
Load more 3664
Load more 3665
Load more 3666
Load more 3667
Load more 3668
Load more 3669
Load more 3670
Load more 3671
Load more 3672
Load more 3673
Load more 3674
Load more 3675
Load more 3676
Load more 3677
Load more 3678
Load more 3679
Load more 3680
Load more 3681
Load more 3682
Load more 3683
Load more 3684
Load more 3685
Load more 3686
Load more 3687
Load more 3688
Load more 3689
Load more 3690
Load more 3691
Load more 3692
Load more 3693
Load more 3694
Load more 3695
Load more 3696
Load more 3697
Load more 3698
Load more 3699
Load more 3700
Load more 3701
Load more 3702
Load more 3703
Load more 3704
Load more 3705
Load more 3706
Load more 3707
Load more 3708
Load more 3709
Load more 3710
Load more 3711
Load more 3712
Load more 3713
Load more 3714
Load more 3715
Load more 3716
Load more 3717
Load more 3718
Load more 3719
Load more 3720
Load more 3721
Load more 3722
Load more 3723
Load more 3724
Load more 3725
Load more 3726
Load more 3727
Load more 3728
Load more 3729
Load more 3730
Load more 3731
Load more 3732
Load more 3733
Load more 3734
Load more 3735
Load more 3736
Load more 3737
Load more 3738
Load more 3739
Load more 3740
Load more 3741
Load more 3742
Load more 3743
Load more 3744
Load more 3745
Load more 3746
Load more 3747
Load more 3748
Load more 3749
Load more 3750
Load more 3751
Load more 3752
Load more 3753
Load more 3754
Load more 3755
Load more 3756
Load more 3757
Load more 3758
Load more 3759
Load more 3760
Load more 3761
Load more 3762
Load more 3763
Load more 3764
Load more 3765
Load more 3766
Load more 3767
Load more 3768
Load more 3769
Load more 3770
Load more 3771
Load more 3772
Load more 3773
Load more 3774
Load more 3775
Load more 3776
Load more 3777
Load more 3778
Load more 3779
Load more 3780
Load more 3781
Load more 3782
Load more 3783
Load more 3784
Load more 3785
Load more 3786
Load more 3787
Load more 3788
Load more 3789
Load more 3790
Load more 3791
Load more 3792
Load more 3793
Load more 3794
Load more 3795
Load more 3796
Load more 3797
Load more 3798
Load more 3799
Load more 3800
Load more 3801
Load more 3802
Load more 3803
Load more 3804
Load more 3805
Load more 3806
Load more 3807
Load more 3808
Load more 3809
Load more 3810
Load more 3811
Load more 3812
Load more 3813
Load more 3814
Load more 3815
Load more 3816
Load more 3817
Load more 3818
Load more 3819
Load more 3820
Load more 3821
Load more 3822
Load more 3823
Load more 3824
Load more 3825
Load more 3826
Load more 3827
Load more 3828
Load more 3829
Load more 3830
Load more 3831
Load more 3832
Load more 3833
Load more 3834
Load more 3835
Load more 3836
Load more 3837
Load more 3838
Load more 3839
Load more 3840
Load more 3841
Load more 3842
Load more 3843
Load more 3844
Load more 3845
Load more 3846
Load more 3847
Load more 3848
Load more 3849
Load more 3850
Load more 3851
Load more 3852
Load more 3853
Load more 3854
Load more 3855
Load more 3856
Load more 3857
Load more 3858
Load more 3859
Load more 3860
Load more 3861
Load more 3862
Load more 3863
Load more 3864
Load more 3865
Load more 3866
Load more 3867
Load more 3868
Load more 3869
Load more 3870
Load more 3871
Load more 3872
Load more 3873
Load more 3874
Load more 3875
Load more 3876
Load more 3877
Load more 3878
Load more 3879
Load more 3880
Load more 3881
Load more 3882
Load more 3883
Load more 3884
Load more 3885
Load more 3886
Load more 3887
Load more 3888
Load more 3889
Load more 3890
Load more 3891
Load more 3892
Load more 3893
Load more 3894
Load more 3895
Load more 3896
Load more 3897
Load more 3898
Load more 3899
Load more 3900
Load more 3901
Load more 3902
Load more 3903
Load more 3904
Load more 3905
Load more 3906
Load more 3907
Load more 3908
Load more 3909
Load more 3910
Load more 3911
Load more 3912
Load more 3913
Load more 3914
Load more 3915
Load more 3916
Load more 3917
Load more 3918
Load more 3919
Load more 3920
Load more 3921
Load more 3922
Load more 3923
Load more 3924
Load more 3925
Load more 3926
Load more 3927
Load more 3928
Load more 3929
Load more 3930
Load more 3931
Load more 3932
Load more 3933
Load more 3934
Load more 3935
Load more 3936
Load more 3937
Load more 3938
Load more 3939
Load more 3940
Load more 3941
Load more 3942
Load more 3943
Load more 3944
Load more 3945
Load more 3946
Load more 3947
Load more 3948
Load more 3949
Load more 3950
Load more 3951
Load more 3952
Load more 3953
Load more 3954
Load more 3955
Load more 3956
Load more 3957
Load more 3958
Load more 3959
Load more 3960
Load more 3961
Load more 3962
Load more 3963
Load more 3964
Load more 3965
Load more 3966
Load more 3967
Load more 3968
Load more 3969
Load more 3970
Load more 3971
Load more 3972
Load more 3973
Load more 3974
Load more 3975
Load more 3976
Load more 3977
Load more 3978
Load more 3979
Load more 3980
Load more 3981
Load more 3982
Load more 3983
Load more 3984
Load more 3985
Load more 3986
Load more 3987
Load more 3988
Load more 3989
Load more 3990
Load more 3991
Load more 3992
Load more 3993
Load more 3994
Load more 3995
Load more 3996
Load more 3997
Load more 3998
Load more 3999
Load more 4000

$ python3 -m pip install ansible
...
No matching distribution found for ansible-core<2.13,>=2.12.0 (from ansible)

# Alternatively:
ERROR: No matching distribution found for ansible-core<2.13,>=2.12.0

$ python -m pip install --user ansible==4.9.0

$ sudo nano /etc/apt/sources.list

deb http://http.us.debian.org/debian/ testing non-free contrib main

$ sudo apt update
$ sudo apt install -y python3.9

$ python3 --version
Python 3.9.9

$ python3 -m pip install ansible

$ python3 -m pip freeze
ansible==5.0.1
ansible-core==2.12.0
cffi==1.15.0

Load more 3001
Load more 3002
Load more 3003
Load more 3004
Load more 3005
Load more 3006
Load more 3007
Load more 3008
Load more 3009
Load more 3010
Load more 3011
Load more 3012
Load more 3013
Load more 3014
Load more 3015
Load more 3016
Load more 3017
Load more 3018
Load more 3019
Load more 3020
Load more 3021
Load more 3022
Load more 3023
Load more 3024
Load more 3025
Load more 3026
Load more 3027
Load more 3028
Load more 3029
Load more 3030
Load more 3031
Load more 3032
Load more 3033
Load more 3034
Load more 3035
Load more 3036
Load more 3037
Load more 3038
Load more 3039
Load more 3040
Load more 3041
Load more 3042
Load more 3043
Load more 3044
Load more 3045
Load more 3046
Load more 3047
Load more 3048
Load more 3049
Load more 3050
Load more 3051
Load more 3052
Load more 3053
Load more 3054
Load more 3055
Load more 3056
Load more 3057
Load more 3058
Load more 3059
Load more 3060
Load more 3061
Load more 3062
Load more 3063
Load more 3064
Load more 3065
Load more 3066
Load more 3067
Load more 3068
Load more 3069
Load more 3070
Load more 3071
Load more 3072
Load more 3073
Load more 3074
Load more 3075
Load more 3076
Load more 3077
Load more 3078
Load more 3079
Load more 3080
Load more 3081
Load more 3082
Load more 3083
Load more 3084
Load more 3085
Load more 3086
Load more 3087
Load more 3088
Load more 3089
Load more 3090
Load more 3091
Load more 3092
Load more 3093
Load more 3094
Load more 3095
Load more 3096
Load more 3097
Load more 3098
Load more 3099
Load more 3100
Load more 3101
Load more 3102
Load more 3103
Load more 3104
Load more 3105
Load more 3106
Load more 3107
Load more 3108
Load more 3109
Load more 3110
Load more 3111
Load more 3112
Load more 3113
Load more 3114
Load more 3115
Load more 3116
Load more 3117
Load more 3118
Load more 3119
Load more 3120
Load more 3121
Load more 3122
Load more 3123
Load more 3124
Load more 3125
Load more 3126
Load more 3127
Load more 3128
Load more 3129
Load more 3130
Load more 3131
Load more 3132
Load more 3133
Load more 3134
Load more 3135
Load more 3136
Load more 3137
Load more 3138
Load more 3139
Load more 3140
Load more 3141
Load more 3142
Load more 3143
Load more 3144
Load more 3145
Load more 3146
Load more 3147
Load more 3148
Load more 3149
Load more 3150
Load more 3151
Load more 3152
Load more 3153
Load more 3154
Load more 3155
Load more 3156
Load more 3157
Load more 3158
Load more 3159
Load more 3160
Load more 3161
Load more 3162
Load more 3163
Load more 3164
Load more 3165
Load more 3166
Load more 3167
Load more 3168
Load more 3169
Load more 3170
Load more 3171
Load more 3172
Load more 3173
Load more 3174
Load more 3175
Load more 3176
Load more 3177
Load more 3178
Load more 3179
Load more 3180
Load more 3181
Load more 3182
Load more 3183
Load more 3184
Load more 3185
Load more 3186
Load more 3187
Load more 3188
Load more 3189
Load more 3190
Load more 3191
Load more 3192
Load more 3193
Load more 3194
Load more 3195
Load more 3196
Load more 3197
Load more 3198
Load more 3199
Load more 3200
Load more 3201
Load more 3202
Load more 3203
Load more 3204
Load more 3205
Load more 3206
Load more 3207
Load more 3208
Load more 3209
Load more 3210
Load more 3211
Load more 3212
Load more 3213
Load more 3214
Load more 3215
Load more 3216
Load more 3217
Load more 3218
Load more 3219
Load more 3220
Load more 3221
Load more 3222
Load more 3223
Load more 3224
Load more 3225
Load more 3226
Load more 3227
Load more 3228
Load more 3229
Load more 3230
Load more 3231
Load more 3232
Load more 3233
Load more 3234
Load more 3235
Load more 3236
Load more 3237
Load more 3238
Load more 3239
Load more 3240
Load more 3241
Load more 3242
Load more 3243
Load more 3244
Load more 3245
Load more 3246
Load more 3247
Load more 3248
Load more 3249
Load more 3250
Load more 3251
Load more 3252
Load more 3253
Load more 3254
Load more 3255
Load more 3256
Load more 3257
Load more 3258
Load more 3259
Load more 3260
Load more 3261
Load more 3262
Load more 3263
Load more 3264
Load more 3265
Load more 3266
Load more 3267
Load more 3268
Load more 3269
Load more 3270
Load more 3271
Load more 3272
Load more 3273
Load more 3274
Load more 3275
Load more 3276
Load more 3277
Load more 3278
Load more 3279
Load more 3280
Load more 3281
Load more 3282
Load more 3283
Load more 3284
Load more 3285
Load more 3286
Load more 3287
Load more 3288
Load more 3289
Load more 3290
Load more 3291
Load more 3292
Load more 3293
Load more 3294
Load more 3295
Load more 3296
Load more 3297
Load more 3298
Load more 3299
Load more 3300
Load more 3301
Load more 3302
Load more 3303
Load more 3304
Load more 3305
Load more 3306
Load more 3307
Load more 3308
Load more 3309
Load more 3310
Load more 3311
Load more 3312
Load more 3313
Load more 3314
Load more 3315
Load more 3316
Load more 3317
Load more 3318
Load more 3319
Load more 3320
Load more 3321
Load more 3322
Load more 3323
Load more 3324
Load more 3325
Load more 3326
Load more 3327
Load more 3328
Load more 3329
Load more 3330
Load more 3331
Load more 3332
Load more 3333
Load more 3334
Load more 3335
Load more 3336
Load more 3337
Load more 3338
Load more 3339
Load more 3340
Load more 3341
Load more 3342
Load more 3343
Load more 3344
Load more 3345
Load more 3346
Load more 3347
Load more 3348
Load more 3349
Load more 3350
Load more 3351
Load more 3352
Load more 3353
Load more 3354
Load more 3355
Load more 3356
Load more 3357
Load more 3358
Load more 3359
Load more 3360
Load more 3361
Load more 3362
Load more 3363
Load more 3364
Load more 3365
Load more 3366
Load more 3367
Load more 3368
Load more 3369
Load more 3370
Load more 3371
Load more 3372
Load more 3373
Load more 3374
Load more 3375
Load more 3376
Load more 3377
Load more 3378
Load more 3379
Load more 3380
Load more 3381
Load more 3382
Load more 3383
Load more 3384
Load more 3385
Load more 3386
Load more 3387
Load more 3388
Load more 3389
Load more 3390
Load more 3391
Load more 3392
Load more 3393
Load more 3394
Load more 3395
Load more 3396
Load more 3397
Load more 3398
Load more 3399
Load more 3400
Load more 3401
Load more 3402
Load more 3403
Load more 3404
Load more 3405
Load more 3406
Load more 3407
Load more 3408
Load more 3409
Load more 3410
Load more 3411
Load more 3412
Load more 3413
Load more 3414
Load more 3415
Load more 3416
Load more 3417
Load more 3418
Load more 3419
Load more 3420
Load more 3421
Load more 3422
Load more 3423
Load more 3424
Load more 3425
Load more 3426
Load more 3427
Load more 3428
Load more 3429
Load more 3430
Load more 3431
Load more 3432
Load more 3433
Load more 3434
Load more 3435
Load more 3436
Load more 3437
Load more 3438
Load more 3439
Load more 3440
Load more 3441
Load more 3442
Load more 3443
Load more 3444
Load more 3445
Load more 3446
Load more 3447
Load more 3448
Load more 3449
Load more 3450
Load more 3451
Load more 3452
Load more 3453
Load more 3454
Load more 3455
Load more 3456
Load more 3457
Load more 3458
Load more 3459
Load more 3460
Load more 3461
Load more 3462
Load more 3463
Load more 3464
Load more 3465
Load more 3466
Load more 3467
Load more 3468
Load more 3469
Load more 3470
Load more 3471
Load more 3472
Load more 3473
Load more 3474
Load more 3475
Load more 3476
Load more 3477
Load more 3478
Load more 3479
Load more 3480
Load more 3481
Load more 3482
Load more 3483
Load more 3484
Load more 3485
Load more 3486
Load more 3487
Load more 3488
Load more 3489
Load more 3490
Load more 3491
Load more 3492
Load more 3493
Load more 3494
Load more 3495
Load more 3496
Load more 3497
Load more 3498
Load more 3499
Load more 3500

Load more 2501
Load more 2502
Load more 2503
Load more 2504
Load more 2505
Load more 2506
Load more 2507
Load more 2508
Load more 2509
Load more 2510
Load more 2511
Load more 2512
Load more 2513
Load more 2514
Load more 2515
Load more 2516
Load more 2517
Load more 2518
Load more 2519
Load more 2520
Load more 2521
Load more 2522
Load more 2523
Load more 2524
Load more 2525
Load more 2526
Load more 2527
Load more 2528
Load more 2529
Load more 2530
Load more 2531
Load more 2532
Load more 2533
Load more 2534
Load more 2535
Load more 2536
Load more 2537
Load more 2538
Load more 2539
Load more 2540
Load more 2541
Load more 2542
Load more 2543
Load more 2544
Load more 2545
Load more 2546
Load more 2547
Load more 2548
Load more 2549
Load more 2550
Load more 2551
Load more 2552
Load more 2553
Load more 2554
Load more 2555
Load more 2556
Load more 2557
Load more 2558
Load more 2559
Load more 2560
Load more 2561
Load more 2562
Load more 2563
Load more 2564
Load more 2565
Load more 2566
Load more 2567
Load more 2568
Load more 2569
Load more 2570
Load more 2571
Load more 2572
Load more 2573
Load more 2574
Load more 2575
Load more 2576
Load more 2577
Load more 2578
Load more 2579
Load more 2580
Load more 2581
Load more 2582
Load more 2583
Load more 2584
Load more 2585
Load more 2586
Load more 2587
Load more 2588
Load more 2589
Load more 2590
Load more 2591
Load more 2592
Load more 2593
Load more 2594
Load more 2595
Load more 2596
Load more 2597
Load more 2598
Load more 2599
Load more 2600
Load more 2601
Load more 2602
Load more 2603
Load more 2604
Load more 2605
Load more 2606
Load more 2607
Load more 2608
Load more 2609
Load more 2610
Load more 2611
Load more 2612
Load more 2613
Load more 2614
Load more 2615
Load more 2616
Load more 2617
Load more 2618
Load more 2619
Load more 2620
Load more 2621
Load more 2622
Load more 2623
Load more 2624
Load more 2625
Load more 2626
Load more 2627
Load more 2628
Load more 2629
Load more 2630
Load more 2631
Load more 2632
Load more 2633
Load more 2634
Load more 2635
Load more 2636
Load more 2637
Load more 2638
Load more 2639
Load more 2640
Load more 2641
Load more 2642
Load more 2643
Load more 2644
Load more 2645
Load more 2646
Load more 2647
Load more 2648
Load more 2649
Load more 2650
Load more 2651
Load more 2652
Load more 2653
Load more 2654
Load more 2655
Load more 2656
Load more 2657
Load more 2658
Load more 2659
Load more 2660
Load more 2661
Load more 2662
Load more 2663
Load more 2664
Load more 2665
Load more 2666
Load more 2667
Load more 2668
Load more 2669
Load more 2670
Load more 2671
Load more 2672
Load more 2673
Load more 2674
Load more 2675
Load more 2676
Load more 2677
Load more 2678
Load more 2679
Load more 2680
Load more 2681
Load more 2682
Load more 2683
Load more 2684
Load more 2685
Load more 2686
Load more 2687
Load more 2688
Load more 2689
Load more 2690
Load more 2691
Load more 2692
Load more 2693
Load more 2694
Load more 2695
Load more 2696
Load more 2697
Load more 2698
Load more 2699
Load more 2700
Load more 2701
Load more 2702
Load more 2703
Load more 2704
Load more 2705
Load more 2706
Load more 2707
Load more 2708
Load more 2709
Load more 2710
Load more 2711
Load more 2712
Load more 2713
Load more 2714
Load more 2715
Load more 2716
Load more 2717
Load more 2718
Load more 2719
Load more 2720
Load more 2721
Load more 2722
Load more 2723
Load more 2724
Load more 2725
Load more 2726
Load more 2727
Load more 2728
Load more 2729
Load more 2730
Load more 2731
Load more 2732
Load more 2733
Load more 2734
Load more 2735
Load more 2736
Load more 2737
Load more 2738
Load more 2739
Load more 2740
Load more 2741
Load more 2742
Load more 2743
Load more 2744
Load more 2745
Load more 2746
Load more 2747
Load more 2748
Load more 2749
Load more 2750
Load more 2751
Load more 2752
Load more 2753
Load more 2754
Load more 2755
Load more 2756
Load more 2757
Load more 2758
Load more 2759
Load more 2760
Load more 2761
Load more 2762
Load more 2763
Load more 2764
Load more 2765
Load more 2766
Load more 2767
Load more 2768
Load more 2769
Load more 2770
Load more 2771
Load more 2772
Load more 2773
Load more 2774
Load more 2775
Load more 2776
Load more 2777
Load more 2778
Load more 2779
Load more 2780
Load more 2781
Load more 2782
Load more 2783
Load more 2784
Load more 2785
Load more 2786
Load more 2787
Load more 2788
Load more 2789
Load more 2790
Load more 2791
Load more 2792
Load more 2793
Load more 2794
Load more 2795
Load more 2796
Load more 2797
Load more 2798
Load more 2799
Load more 2800
Load more 2801
Load more 2802
Load more 2803
Load more 2804
Load more 2805
Load more 2806
Load more 2807
Load more 2808
Load more 2809
Load more 2810
Load more 2811
Load more 2812
Load more 2813
Load more 2814
Load more 2815
Load more 2816
Load more 2817
Load more 2818
Load more 2819
Load more 2820
Load more 2821
Load more 2822
Load more 2823
Load more 2824
Load more 2825
Load more 2826
Load more 2827
Load more 2828
Load more 2829
Load more 2830
Load more 2831
Load more 2832
Load more 2833
Load more 2834
Load more 2835
Load more 2836
Load more 2837
Load more 2838
Load more 2839
Load more 2840
Load more 2841
Load more 2842
Load more 2843
Load more 2844
Load more 2845
Load more 2846
Load more 2847
Load more 2848
Load more 2849
Load more 2850
Load more 2851
Load more 2852
Load more 2853
Load more 2854
Load more 2855
Load more 2856
Load more 2857
Load more 2858
Load more 2859
Load more 2860
Load more 2861
Load more 2862
Load more 2863
Load more 2864
Load more 2865
Load more 2866
Load more 2867
Load more 2868
Load more 2869
Load more 2870
Load more 2871
Load more 2872
Load more 2873
Load more 2874
Load more 2875
Load more 2876
Load more 2877
Load more 2878
Load more 2879
Load more 2880
Load more 2881
Load more 2882
Load more 2883
Load more 2884
Load more 2885
Load more 2886
Load more 2887
Load more 2888
Load more 2889
Load more 2890
Load more 2891
Load more 2892
Load more 2893
Load more 2894
Load more 2895
Load more 2896
Load more 2897
Load more 2898
Load more 2899
Load more 2900
Load more 2901
Load more 2902
Load more 2903
Load more 2904
Load more 2905
Load more 2906
Load more 2907
Load more 2908
Load more 2909
Load more 2910
Load more 2911
Load more 2912
Load more 2913
Load more 2914
Load more 2915
Load more 2916
Load more 2917
Load more 2918
Load more 2919
Load more 2920
Load more 2921
Load more 2922
Load more 2923
Load more 2924
Load more 2925
Load more 2926
Load more 2927
Load more 2928
Load more 2929
Load more 2930
Load more 2931
Load more 2932
Load more 2933
Load more 2934
Load more 2935
Load more 2936
Load more 2937
Load more 2938
Load more 2939
Load more 2940
Load more 2941
Load more 2942
Load more 2943
Load more 2944
Load more 2945
Load more 2946
Load more 2947
Load more 2948
Load more 2949
Load more 2950
Load more 2951
Load more 2952
Load more 2953
Load more 2954
Load more 2955
Load more 2956
Load more 2957
Load more 2958
Load more 2959
Load more 2960
Load more 2961
Load more 2962
Load more 2963
Load more 2964
Load more 2965
Load more 2966
Load more 2967
Load more 2968
Load more 2969
Load more 2970
Load more 2971
Load more 2972
Load more 2973
Load more 2974
Load more 2975
Load more 2976
Load more 2977
Load more 2978
Load more 2979
Load more 2980
Load more 2981
Load more 2982
Load more 2983
Load more 2984
Load more 2985
Load more 2986
Load more 2987
Load more 2988
Load more 2989
Load more 2990
Load more 2991
Load more 2992
Load more 2993
Load more 2994
Load more 2995
Load more 2996
Load more 2997
Load more 2998
Load more 2999
Load more 3000

This is a simple guide, part of a series I’ll call ‘How-To Guide Without Ads’. In it, I’m going to document how I set up bcache on a Raspberry Pi, so I could use an SSD as a cache in front of a RAID array.

Getting bcache

bcache is sometimes used on Linux devices to allow a more efficient SSD cache to run in front of a single or multiple slower hard drives—typically in a storage array.

In my case, I have three SATA hard drives: /dev/sda, /dev/sdb, and /dev/sdc. And I have one NVMe SSD: /dev/nvme0n1.

I created a RAID5 array with mdadm for the three hard drives, and had the raid device /dev/md0.

I then installed bcache-tools:

$ sudo apt-get install bcache-tools

And used make-bcache to create the backing and cache devices:

$ sudo make-bcache -B /dev/md0
UUID:           eb360a2d-4c62-451d-8549-a68621c633e5
Set UUID:       c8b5c63c-0a44-49f3-bb65-cd4df9b751a0
version:        1
block_size:     1
data_offset:        16

$ sudo make-bcache -C /dev/nvme0n1
UUID:           15bf54e9-be21-4478-b676-a08dad937963
Set UUID:       dea419ba-d795-4566-b01f-bb57fa96eb21
version:        0
nbuckets:       15261770
block_size:     1
bucket_size:        1024
nr_in_set:      1
nr_this_dev:        0
first_bucket:       1

Then I tried to look in /sys/block/md0/bcache/ so I could attach the cache to the backing device, but I realized bcache isn’t loaded into the default Raspberry Pi OS kernel… so I’ll have to compile that in.

Getting bcache on Raspberry Pi OS

I cross-compiled the Raspberry Pi Linux kernel, and when I did it, during the menuconfig portion, I selected the following option:

> Device Drivers
  > Multiple devices driver support (RAID and LVM)
    > Block device as cache (BCACHE)

I recompiled the kernel and copied my updated kernel to the Pi, then rebooted.

At this point, I could see the bcache0 device was working:

pi@omv:~ $ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda           8:0    0  3.6T  0 disk  
└─md0         9:0    0  7.3T  0 raid5 
  └─bcache0 254:0    0  7.3T  0 disk  /mnt
sdb           8:16   0  3.6T  0 disk  
└─md0         9:0    0  7.3T  0 raid5 
  └─bcache0 254:0    0  7.3T  0 disk  /mnt
sdc           8:32   0  3.6T  0 disk  
└─md0         9:0    0  7.3T  0 raid5 
  └─bcache0 254:0    0  7.3T  0 disk  /mnt
mmcblk0     179:0    0 14.8G  0 disk  
├─mmcblk0p1 179:1    0  256M  0 part  /boot
└─mmcblk0p2 179:2    0 14.6G  0 part  /
nvme0n1     259:0    0  7.3T  0 disk  

But if I checked on the status of the cache, it said there was no cache:

pi@omv:~ $ cat /sys/block/bcache0/bcache/state
no cache

Attaching the SSD cache to the backing device

Finally, it’s time to attach the SSD cache to the backing device:

$ sudo su  # switch to the root user
# cd /sys/block/md0/bcache/
# echo dea419ba-d795-4566-b01f-bb57fa96eb21 > attach
# cat state 
clean

The UUID in the echo command above comes from the ‘Set UUID’ output from the make-bcache -C command earlier.

Creating a filesystem and mounting

To actually use the device, I formatted it and mounted it to /mnt:

$ sudo mkfs.ext4 -E lazy_itable_init=0,lazy_journal_init=0 /dev/bcache0
$ sudo mount /dev/bcache0 /mnt

To avoid the initialization when making the filesystem, you can omit the -E option entirely. But for RAID arrays I typically let it go full blast on first initialization, because I don’t like relying on ext4lazyinit on a RAID array—it can take days at its reduced rate, and affect RAID performance that whole time!

Getting stats

You can check the stats from bcache with:

$ tail /sys/block/bcache0/bcache/stats_total/*
==> /sys/block/bcache0/bcache/stats_total/bypassed <==
563.1M

==> /sys/block/bcache0/bcache/stats_total/cache_bypass_hits <==
0

==> /sys/block/bcache0/bcache/stats_total/cache_bypass_misses <==
0

==> /sys/block/bcache0/bcache/stats_total/cache_hit_ratio <==
13

==> /sys/block/bcache0/bcache/stats_total/cache_hits <==
132

==> /sys/block/bcache0/bcache/stats_total/cache_miss_collisions <==
0

==> /sys/block/bcache0/bcache/stats_total/cache_misses <==
849

==> /sys/block/bcache0/bcache/stats_total/cache_readaheads <==
0

Switching the caching mode

There are multiple caching modes, including writeback, writethrough, writearound, and none. The most performant (but most dangerous, especially if you’re using a single SSD and not a set of SSDs in RAID 1 for safety) is writeback, which caches reads, and writes data to the SSD first (considering a write ‘complete’ once written to the SSD), then asynchronously copies that data to the backing device.

Load more 2001
Load more 2002
Load more 2003
Load more 2004
Load more 2005
Load more 2006
Load more 2007
Load more 2008
Load more 2009
Load more 2010
Load more 2011
Load more 2012
Load more 2013
Load more 2014
Load more 2015
Load more 2016
Load more 2017
Load more 2018
Load more 2019
Load more 2020
Load more 2021
Load more 2022
Load more 2023
Load more 2024
Load more 2025
Load more 2026
Load more 2027
Load more 2028
Load more 2029
Load more 2030
Load more 2031
Load more 2032
Load more 2033
Load more 2034
Load more 2035
Load more 2036
Load more 2037
Load more 2038
Load more 2039
Load more 2040
Load more 2041
Load more 2042
Load more 2043
Load more 2044
Load more 2045
Load more 2046
Load more 2047
Load more 2048
Load more 2049
Load more 2050
Load more 2051
Load more 2052
Load more 2053
Load more 2054
Load more 2055
Load more 2056
Load more 2057
Load more 2058
Load more 2059
Load more 2060
Load more 2061
Load more 2062
Load more 2063
Load more 2064
Load more 2065
Load more 2066
Load more 2067
Load more 2068
Load more 2069
Load more 2070
Load more 2071
Load more 2072
Load more 2073
Load more 2074
Load more 2075
Load more 2076
Load more 2077
Load more 2078
Load more 2079
Load more 2080
Load more 2081
Load more 2082
Load more 2083
Load more 2084
Load more 2085
Load more 2086
Load more 2087
Load more 2088
Load more 2089
Load more 2090
Load more 2091
Load more 2092
Load more 2093
Load more 2094
Load more 2095
Load more 2096
Load more 2097
Load more 2098
Load more 2099
Load more 2100
Load more 2101
Load more 2102
Load more 2103
Load more 2104
Load more 2105
Load more 2106
Load more 2107
Load more 2108
Load more 2109
Load more 2110
Load more 2111
Load more 2112
Load more 2113
Load more 2114
Load more 2115
Load more 2116
Load more 2117
Load more 2118
Load more 2119
Load more 2120
Load more 2121
Load more 2122
Load more 2123
Load more 2124
Load more 2125
Load more 2126
Load more 2127
Load more 2128
Load more 2129
Load more 2130
Load more 2131
Load more 2132
Load more 2133
Load more 2134
Load more 2135
Load more 2136
Load more 2137
Load more 2138
Load more 2139
Load more 2140
Load more 2141
Load more 2142
Load more 2143
Load more 2144
Load more 2145
Load more 2146
Load more 2147
Load more 2148
Load more 2149
Load more 2150
Load more 2151
Load more 2152
Load more 2153
Load more 2154
Load more 2155
Load more 2156
Load more 2157
Load more 2158
Load more 2159
Load more 2160
Load more 2161
Load more 2162
Load more 2163
Load more 2164
Load more 2165
Load more 2166
Load more 2167
Load more 2168
Load more 2169
Load more 2170
Load more 2171
Load more 2172
Load more 2173
Load more 2174
Load more 2175
Load more 2176
Load more 2177
Load more 2178
Load more 2179
Load more 2180
Load more 2181
Load more 2182
Load more 2183
Load more 2184
Load more 2185
Load more 2186
Load more 2187
Load more 2188
Load more 2189
Load more 2190
Load more 2191
Load more 2192
Load more 2193
Load more 2194
Load more 2195
Load more 2196
Load more 2197
Load more 2198
Load more 2199
Load more 2200
Load more 2201
Load more 2202
Load more 2203
Load more 2204
Load more 2205
Load more 2206
Load more 2207
Load more 2208
Load more 2209
Load more 2210
Load more 2211
Load more 2212
Load more 2213
Load more 2214
Load more 2215
Load more 2216
Load more 2217
Load more 2218
Load more 2219
Load more 2220
Load more 2221
Load more 2222
Load more 2223
Load more 2224
Load more 2225
Load more 2226
Load more 2227
Load more 2228
Load more 2229
Load more 2230
Load more 2231
Load more 2232
Load more 2233
Load more 2234
Load more 2235
Load more 2236
Load more 2237
Load more 2238
Load more 2239
Load more 2240
Load more 2241
Load more 2242
Load more 2243
Load more 2244
Load more 2245
Load more 2246
Load more 2247
Load more 2248
Load more 2249
Load more 2250
Load more 2251
Load more 2252
Load more 2253
Load more 2254
Load more 2255
Load more 2256
Load more 2257
Load more 2258
Load more 2259
Load more 2260
Load more 2261
Load more 2262
Load more 2263
Load more 2264
Load more 2265
Load more 2266
Load more 2267
Load more 2268
Load more 2269
Load more 2270
Load more 2271
Load more 2272
Load more 2273
Load more 2274
Load more 2275
Load more 2276
Load more 2277
Load more 2278
Load more 2279
Load more 2280
Load more 2281
Load more 2282
Load more 2283
Load more 2284
Load more 2285
Load more 2286
Load more 2287
Load more 2288
Load more 2289
Load more 2290
Load more 2291
Load more 2292
Load more 2293
Load more 2294
Load more 2295
Load more 2296
Load more 2297
Load more 2298
Load more 2299
Load more 2300
Load more 2301
Load more 2302
Load more 2303
Load more 2304
Load more 2305
Load more 2306
Load more 2307
Load more 2308
Load more 2309
Load more 2310
Load more 2311
Load more 2312
Load more 2313
Load more 2314
Load more 2315
Load more 2316
Load more 2317
Load more 2318
Load more 2319
Load more 2320
Load more 2321
Load more 2322
Load more 2323
Load more 2324
Load more 2325
Load more 2326
Load more 2327
Load more 2328
Load more 2329
Load more 2330
Load more 2331
Load more 2332
Load more 2333
Load more 2334
Load more 2335
Load more 2336
Load more 2337
Load more 2338
Load more 2339
Load more 2340
Load more 2341
Load more 2342
Load more 2343
Load more 2344
Load more 2345
Load more 2346
Load more 2347
Load more 2348
Load more 2349
Load more 2350
Load more 2351
Load more 2352
Load more 2353
Load more 2354
Load more 2355
Load more 2356
Load more 2357
Load more 2358
Load more 2359
Load more 2360
Load more 2361
Load more 2362
Load more 2363
Load more 2364
Load more 2365
Load more 2366
Load more 2367
Load more 2368
Load more 2369
Load more 2370
Load more 2371
Load more 2372
Load more 2373
Load more 2374
Load more 2375
Load more 2376
Load more 2377
Load more 2378
Load more 2379
Load more 2380
Load more 2381
Load more 2382
Load more 2383
Load more 2384
Load more 2385
Load more 2386
Load more 2387
Load more 2388
Load more 2389
Load more 2390
Load more 2391
Load more 2392
Load more 2393
Load more 2394
Load more 2395
Load more 2396
Load more 2397
Load more 2398
Load more 2399
Load more 2400
Load more 2401
Load more 2402
Load more 2403
Load more 2404
Load more 2405
Load more 2406
Load more 2407
Load more 2408
Load more 2409
Load more 2410
Load more 2411
Load more 2412
Load more 2413
Load more 2414
Load more 2415
Load more 2416
Load more 2417
Load more 2418
Load more 2419
Load more 2420
Load more 2421
Load more 2422
Load more 2423
Load more 2424
Load more 2425
Load more 2426
Load more 2427
Load more 2428
Load more 2429
Load more 2430
Load more 2431
Load more 2432
Load more 2433
Load more 2434
Load more 2435
Load more 2436
Load more 2437
Load more 2438
Load more 2439
Load more 2440
Load more 2441
Load more 2442
Load more 2443
Load more 2444
Load more 2445
Load more 2446
Load more 2447
Load more 2448
Load more 2449
Load more 2450
Load more 2451
Load more 2452
Load more 2453
Load more 2454
Load more 2455
Load more 2456
Load more 2457
Load more 2458
Load more 2459
Load more 2460
Load more 2461
Load more 2462
Load more 2463
Load more 2464
Load more 2465
Load more 2466
Load more 2467
Load more 2468
Load more 2469
Load more 2470
Load more 2471
Load more 2472
Load more 2473
Load more 2474
Load more 2475
Load more 2476
Load more 2477
Load more 2478
Load more 2479
Load more 2480
Load more 2481
Load more 2482
Load more 2483
Load more 2484
Load more 2485
Load more 2486
Load more 2487
Load more 2488
Load more 2489
Load more 2490
Load more 2491
Load more 2492
Load more 2493
Load more 2494
Load more 2495
Load more 2496
Load more 2497
Load more 2498
Load more 2499
Load more 2500

$ sudo apt-get install -y raspberrypi-kernel-headers
$ tar vjxf r8125-9.007.01.tar.bz2
$ cd r8125-9.007.01/
$ sudo ./autorun.sh

Load more 1501
Load more 1502
Load more 1503
Load more 1504
Load more 1505
Load more 1506
Load more 1507
Load more 1508
Load more 1509
Load more 1510
Load more 1511
Load more 1512
Load more 1513
Load more 1514
Load more 1515
Load more 1516
Load more 1517
Load more 1518
Load more 1519
Load more 1520
Load more 1521
Load more 1522
Load more 1523
Load more 1524
Load more 1525
Load more 1526
Load more 1527
Load more 1528
Load more 1529
Load more 1530
Load more 1531
Load more 1532
Load more 1533
Load more 1534
Load more 1535
Load more 1536
Load more 1537
Load more 1538
Load more 1539
Load more 1540
Load more 1541
Load more 1542
Load more 1543
Load more 1544
Load more 1545
Load more 1546
Load more 1547
Load more 1548
Load more 1549
Load more 1550
Load more 1551
Load more 1552
Load more 1553
Load more 1554
Load more 1555
Load more 1556
Load more 1557
Load more 1558
Load more 1559
Load more 1560
Load more 1561
Load more 1562
Load more 1563
Load more 1564
Load more 1565
Load more 1566
Load more 1567
Load more 1568
Load more 1569
Load more 1570
Load more 1571
Load more 1572
Load more 1573
Load more 1574
Load more 1575
Load more 1576
Load more 1577
Load more 1578
Load more 1579
Load more 1580
Load more 1581
Load more 1582
Load more 1583
Load more 1584
Load more 1585
Load more 1586
Load more 1587
Load more 1588
Load more 1589
Load more 1590
Load more 1591
Load more 1592
Load more 1593
Load more 1594
Load more 1595
Load more 1596
Load more 1597
Load more 1598
Load more 1599
Load more 1600
Load more 1601
Load more 1602
Load more 1603
Load more 1604
Load more 1605
Load more 1606
Load more 1607
Load more 1608
Load more 1609
Load more 1610
Load more 1611
Load more 1612
Load more 1613
Load more 1614
Load more 1615
Load more 1616
Load more 1617
Load more 1618
Load more 1619
Load more 1620
Load more 1621
Load more 1622
Load more 1623
Load more 1624
Load more 1625
Load more 1626
Load more 1627
Load more 1628
Load more 1629
Load more 1630
Load more 1631
Load more 1632
Load more 1633
Load more 1634
Load more 1635
Load more 1636
Load more 1637
Load more 1638
Load more 1639
Load more 1640
Load more 1641
Load more 1642
Load more 1643
Load more 1644
Load more 1645
Load more 1646
Load more 1647
Load more 1648
Load more 1649
Load more 1650
Load more 1651
Load more 1652
Load more 1653
Load more 1654
Load more 1655
Load more 1656
Load more 1657
Load more 1658
Load more 1659
Load more 1660
Load more 1661
Load more 1662
Load more 1663
Load more 1664
Load more 1665
Load more 1666
Load more 1667
Load more 1668
Load more 1669
Load more 1670
Load more 1671
Load more 1672
Load more 1673
Load more 1674
Load more 1675
Load more 1676
Load more 1677
Load more 1678
Load more 1679
Load more 1680
Load more 1681
Load more 1682
Load more 1683
Load more 1684
Load more 1685
Load more 1686
Load more 1687
Load more 1688
Load more 1689
Load more 1690
Load more 1691
Load more 1692
Load more 1693
Load more 1694
Load more 1695
Load more 1696
Load more 1697
Load more 1698
Load more 1699
Load more 1700
Load more 1701
Load more 1702
Load more 1703
Load more 1704
Load more 1705
Load more 1706
Load more 1707
Load more 1708
Load more 1709
Load more 1710
Load more 1711
Load more 1712
Load more 1713
Load more 1714
Load more 1715
Load more 1716
Load more 1717
Load more 1718
Load more 1719
Load more 1720
Load more 1721
Load more 1722
Load more 1723
Load more 1724
Load more 1725
Load more 1726
Load more 1727
Load more 1728
Load more 1729
Load more 1730
Load more 1731
Load more 1732
Load more 1733
Load more 1734
Load more 1735
Load more 1736
Load more 1737
Load more 1738
Load more 1739
Load more 1740
Load more 1741
Load more 1742
Load more 1743
Load more 1744
Load more 1745
Load more 1746
Load more 1747
Load more 1748
Load more 1749
Load more 1750
Load more 1751
Load more 1752
Load more 1753
Load more 1754
Load more 1755
Load more 1756
Load more 1757
Load more 1758
Load more 1759
Load more 1760
Load more 1761
Load more 1762
Load more 1763
Load more 1764
Load more 1765
Load more 1766
Load more 1767
Load more 1768
Load more 1769
Load more 1770
Load more 1771
Load more 1772
Load more 1773
Load more 1774
Load more 1775
Load more 1776
Load more 1777
Load more 1778
Load more 1779
Load more 1780
Load more 1781
Load more 1782
Load more 1783
Load more 1784
Load more 1785
Load more 1786
Load more 1787
Load more 1788
Load more 1789
Load more 1790
Load more 1791
Load more 1792
Load more 1793
Load more 1794
Load more 1795
Load more 1796
Load more 1797
Load more 1798
Load more 1799
Load more 1800
Load more 1801
Load more 1802
Load more 1803
Load more 1804
Load more 1805
Load more 1806
Load more 1807
Load more 1808
Load more 1809
Load more 1810
Load more 1811
Load more 1812
Load more 1813
Load more 1814
Load more 1815
Load more 1816
Load more 1817
Load more 1818
Load more 1819
Load more 1820
Load more 1821
Load more 1822
Load more 1823
Load more 1824
Load more 1825
Load more 1826
Load more 1827
Load more 1828
Load more 1829
Load more 1830
Load more 1831
Load more 1832
Load more 1833
Load more 1834
Load more 1835
Load more 1836
Load more 1837
Load more 1838
Load more 1839
Load more 1840
Load more 1841
Load more 1842
Load more 1843
Load more 1844
Load more 1845
Load more 1846
Load more 1847
Load more 1848
Load more 1849
Load more 1850
Load more 1851
Load more 1852
Load more 1853
Load more 1854
Load more 1855
Load more 1856
Load more 1857
Load more 1858
Load more 1859
Load more 1860
Load more 1861
Load more 1862
Load more 1863
Load more 1864
Load more 1865
Load more 1866
Load more 1867
Load more 1868
Load more 1869
Load more 1870
Load more 1871
Load more 1872
Load more 1873
Load more 1874
Load more 1875
Load more 1876
Load more 1877
Load more 1878
Load more 1879
Load more 1880
Load more 1881
Load more 1882
Load more 1883
Load more 1884
Load more 1885
Load more 1886
Load more 1887
Load more 1888
Load more 1889
Load more 1890
Load more 1891
Load more 1892
Load more 1893
Load more 1894
Load more 1895
Load more 1896
Load more 1897
Load more 1898
Load more 1899
Load more 1900
Load more 1901
Load more 1902
Load more 1903
Load more 1904
Load more 1905
Load more 1906
Load more 1907
Load more 1908
Load more 1909
Load more 1910
Load more 1911
Load more 1912
Load more 1913
Load more 1914
Load more 1915
Load more 1916
Load more 1917
Load more 1918
Load more 1919
Load more 1920
Load more 1921
Load more 1922
Load more 1923
Load more 1924
Load more 1925
Load more 1926
Load more 1927
Load more 1928
Load more 1929
Load more 1930
Load more 1931
Load more 1932
Load more 1933
Load more 1934
Load more 1935
Load more 1936
Load more 1937
Load more 1938
Load more 1939
Load more 1940
Load more 1941
Load more 1942
Load more 1943
Load more 1944
Load more 1945
Load more 1946
Load more 1947
Load more 1948
Load more 1949
Load more 1950
Load more 1951
Load more 1952
Load more 1953
Load more 1954
Load more 1955
Load more 1956
Load more 1957
Load more 1958
Load more 1959
Load more 1960
Load more 1961
Load more 1962
Load more 1963
Load more 1964
Load more 1965
Load more 1966
Load more 1967
Load more 1968
Load more 1969
Load more 1970
Load more 1971
Load more 1972
Load more 1973
Load more 1974
Load more 1975
Load more 1976
Load more 1977
Load more 1978
Load more 1979
Load more 1980
Load more 1981
Load more 1982
Load more 1983
Load more 1984
Load more 1985
Load more 1986
Load more 1987
Load more 1988
Load more 1989
Load more 1990
Load more 1991
Load more 1992
Load more 1993
Load more 1994
Load more 1995
Load more 1996
Load more 1997
Load more 1998
Load more 1999
Load more 2000

Load more 1001
Load more 1002
Load more 1003
Load more 1004
Load more 1005
Load more 1006
Load more 1007
Load more 1008
Load more 1009
Load more 1010
Load more 1011
Load more 1012
Load more 1013
Load more 1014
Load more 1015
Load more 1016
Load more 1017
Load more 1018
Load more 1019
Load more 1020
Load more 1021
Load more 1022
Load more 1023
Load more 1024
Load more 1025
Load more 1026
Load more 1027
Load more 1028
Load more 1029
Load more 1030
Load more 1031
Load more 1032
Load more 1033
Load more 1034
Load more 1035
Load more 1036
Load more 1037
Load more 1038
Load more 1039
Load more 1040
Load more 1041
Load more 1042
Load more 1043
Load more 1044
Load more 1045
Load more 1046
Load more 1047
Load more 1048
Load more 1049
Load more 1050
Load more 1051
Load more 1052
Load more 1053
Load more 1054
Load more 1055
Load more 1056
Load more 1057
Load more 1058
Load more 1059
Load more 1060
Load more 1061
Load more 1062
Load more 1063
Load more 1064
Load more 1065
Load more 1066
Load more 1067
Load more 1068
Load more 1069
Load more 1070
Load more 1071
Load more 1072
Load more 1073
Load more 1074
Load more 1075
Load more 1076
Load more 1077
Load more 1078
Load more 1079
Load more 1080
Load more 1081
Load more 1082
Load more 1083
Load more 1084
Load more 1085
Load more 1086
Load more 1087
Load more 1088
Load more 1089
Load more 1090
Load more 1091
Load more 1092
Load more 1093
Load more 1094
Load more 1095
Load more 1096
Load more 1097
Load more 1098
Load more 1099
Load more 1100
Load more 1101
Load more 1102
Load more 1103
Load more 1104
Load more 1105
Load more 1106
Load more 1107
Load more 1108
Load more 1109
Load more 1110
Load more 1111
Load more 1112
Load more 1113
Load more 1114
Load more 1115
Load more 1116
Load more 1117
Load more 1118
Load more 1119
Load more 1120
Load more 1121
Load more 1122
Load more 1123
Load more 1124
Load more 1125
Load more 1126
Load more 1127
Load more 1128
Load more 1129
Load more 1130
Load more 1131
Load more 1132
Load more 1133
Load more 1134
Load more 1135
Load more 1136
Load more 1137
Load more 1138
Load more 1139
Load more 1140
Load more 1141
Load more 1142
Load more 1143
Load more 1144
Load more 1145
Load more 1146
Load more 1147
Load more 1148
Load more 1149
Load more 1150
Load more 1151
Load more 1152
Load more 1153
Load more 1154
Load more 1155
Load more 1156
Load more 1157
Load more 1158
Load more 1159
Load more 1160
Load more 1161
Load more 1162
Load more 1163
Load more 1164
Load more 1165
Load more 1166
Load more 1167
Load more 1168
Load more 1169
Load more 1170
Load more 1171
Load more 1172
Load more 1173
Load more 1174
Load more 1175
Load more 1176
Load more 1177
Load more 1178
Load more 1179
Load more 1180
Load more 1181
Load more 1182
Load more 1183
Load more 1184
Load more 1185
Load more 1186
Load more 1187
Load more 1188
Load more 1189
Load more 1190
Load more 1191
Load more 1192
Load more 1193
Load more 1194
Load more 1195
Load more 1196
Load more 1197
Load more 1198
Load more 1199
Load more 1200
Load more 1201
Load more 1202
Load more 1203
Load more 1204
Load more 1205
Load more 1206
Load more 1207
Load more 1208
Load more 1209
Load more 1210
Load more 1211
Load more 1212
Load more 1213
Load more 1214
Load more 1215
Load more 1216
Load more 1217
Load more 1218
Load more 1219
Load more 1220
Load more 1221
Load more 1222
Load more 1223
Load more 1224
Load more 1225
Load more 1226
Load more 1227
Load more 1228
Load more 1229
Load more 1230
Load more 1231
Load more 1232
Load more 1233
Load more 1234
Load more 1235
Load more 1236
Load more 1237
Load more 1238
Load more 1239
Load more 1240
Load more 1241
Load more 1242
Load more 1243
Load more 1244
Load more 1245
Load more 1246
Load more 1247
Load more 1248
Load more 1249
Load more 1250
Load more 1251
Load more 1252
Load more 1253
Load more 1254
Load more 1255
Load more 1256
Load more 1257
Load more 1258
Load more 1259
Load more 1260
Load more 1261
Load more 1262
Load more 1263
Load more 1264
Load more 1265
Load more 1266
Load more 1267
Load more 1268
Load more 1269
Load more 1270
Load more 1271
Load more 1272
Load more 1273
Load more 1274
Load more 1275
Load more 1276
Load more 1277
Load more 1278
Load more 1279
Load more 1280
Load more 1281
Load more 1282
Load more 1283
Load more 1284
Load more 1285
Load more 1286
Load more 1287
Load more 1288
Load more 1289
Load more 1290
Load more 1291
Load more 1292
Load more 1293
Load more 1294
Load more 1295
Load more 1296
Load more 1297
Load more 1298
Load more 1299
Load more 1300
Load more 1301
Load more 1302
Load more 1303
Load more 1304
Load more 1305
Load more 1306
Load more 1307
Load more 1308
Load more 1309
Load more 1310
Load more 1311
Load more 1312
Load more 1313
Load more 1314
Load more 1315
Load more 1316
Load more 1317
Load more 1318
Load more 1319
Load more 1320
Load more 1321
Load more 1322
Load more 1323
Load more 1324
Load more 1325
Load more 1326
Load more 1327
Load more 1328
Load more 1329
Load more 1330
Load more 1331
Load more 1332
Load more 1333
Load more 1334
Load more 1335
Load more 1336
Load more 1337
Load more 1338
Load more 1339
Load more 1340
Load more 1341
Load more 1342
Load more 1343
Load more 1344
Load more 1345
Load more 1346
Load more 1347
Load more 1348
Load more 1349
Load more 1350
Load more 1351
Load more 1352
Load more 1353
Load more 1354
Load more 1355
Load more 1356
Load more 1357
Load more 1358
Load more 1359
Load more 1360
Load more 1361
Load more 1362
Load more 1363
Load more 1364
Load more 1365
Load more 1366
Load more 1367
Load more 1368
Load more 1369
Load more 1370
Load more 1371
Load more 1372
Load more 1373
Load more 1374
Load more 1375
Load more 1376
Load more 1377
Load more 1378
Load more 1379
Load more 1380
Load more 1381
Load more 1382
Load more 1383
Load more 1384
Load more 1385
Load more 1386
Load more 1387
Load more 1388
Load more 1389
Load more 1390
Load more 1391
Load more 1392
Load more 1393
Load more 1394
Load more 1395
Load more 1396
Load more 1397
Load more 1398
Load more 1399
Load more 1400
Load more 1401
Load more 1402
Load more 1403
Load more 1404
Load more 1405
Load more 1406
Load more 1407
Load more 1408
Load more 1409
Load more 1410
Load more 1411
Load more 1412
Load more 1413
Load more 1414
Load more 1415
Load more 1416
Load more 1417
Load more 1418
Load more 1419
Load more 1420
Load more 1421
Load more 1422
Load more 1423
Load more 1424
Load more 1425
Load more 1426
Load more 1427
Load more 1428
Load more 1429
Load more 1430
Load more 1431
Load more 1432
Load more 1433
Load more 1434
Load more 1435
Load more 1436
Load more 1437
Load more 1438
Load more 1439
Load more 1440
Load more 1441
Load more 1442
Load more 1443
Load more 1444
Load more 1445
Load more 1446
Load more 1447
Load more 1448
Load more 1449
Load more 1450
Load more 1451
Load more 1452
Load more 1453
Load more 1454
Load more 1455
Load more 1456
Load more 1457
Load more 1458
Load more 1459
Load more 1460
Load more 1461
Load more 1462
Load more 1463
Load more 1464
Load more 1465
Load more 1466
Load more 1467
Load more 1468
Load more 1469
Load more 1470
Load more 1471
Load more 1472
Load more 1473
Load more 1474
Load more 1475
Load more 1476
Load more 1477
Load more 1478
Load more 1479
Load more 1480
Load more 1481
Load more 1482
Load more 1483
Load more 1484
Load more 1485
Load more 1486
Load more 1487
Load more 1488
Load more 1489
Load more 1490
Load more 1491
Load more 1492
Load more 1493
Load more 1494
Load more 1495
Load more 1496
Load more 1497
Load more 1498
Load more 1499
Load more 1500