# ๐Ÿš— ์ฐจ๊ทผ์ฐจ๊ทผ ๋™์‹œ์„ฑ ์ด์Šˆ ํ•ด๊ฒฐํ•˜๊ธฐ (5)

๐Ÿš— ์ฐจ๊ทผ์ฐจ๊ทผ ๋™์‹œ์„ฑ ์ด์Šˆ ํ•ด๊ฒฐํ•˜๊ธฐ (1) - synchronized (opens new window)
๐Ÿš— ์ฐจ๊ทผ์ฐจ๊ทผ ๋™์‹œ์„ฑ ์ด์Šˆ ํ•ด๊ฒฐํ•˜๊ธฐ (2) - Pessimistic Lock (opens new window)
๐Ÿš— ์ฐจ๊ทผ์ฐจ๊ทผ ๋™์‹œ์„ฑ ์ด์Šˆ ํ•ด๊ฒฐํ•˜๊ธฐ (3) - Optimistic Lock (opens new window)
๐Ÿš— ์ฐจ๊ทผ์ฐจ๊ทผ ๋™์‹œ์„ฑ ์ด์Šˆ ํ•ด๊ฒฐํ•˜๊ธฐ (4) - Named Lock (opens new window)
๐Ÿ‘‰ ๐Ÿš— ์ฐจ๊ทผ์ฐจ๊ทผ ๋™์‹œ์„ฑ ์ด์Šˆ ํ•ด๊ฒฐํ•˜๊ธฐ (5) - Spinlock (opens new window)

์ž‘์„ฑ์— ์‚ฌ์šฉ๋œ ์˜ˆ์ œ ์ฝ”๋“œ๋Š” concurrency-spinlock (opens new window)์—์„œ ํ™•์ธํ•ด๋ณผ ์ˆ˜ ์žˆ๋‹ค.

# ์Šคํ•€๋ฝ

์ด๋ฒˆ ์‹œ๊ฐ„์—๋Š” Redis๋ฅผ ํ™œ์šฉํ•œ ์Šคํ•€๋ฝ ๋ฐฉ์‹์œผ๋กœ ๋™์‹œ์„ฑ ์ด์Šˆ๋ฅผ ํ•ด๊ฒฐํ•ด๋ณด๋ ค ํ•œ๋‹ค. ์Šคํ•€๋ฝ์€ Redis์˜ SETNX ๋ช…๋ น์–ด๋ฅผ ํ†ตํ•ด ๋ถ„์‚ฐ๋ฝ์„ ๊ตฌํ˜„ํ•  ์ˆ˜ ์žˆ๋‹ค.

SETNX๋Š” SET if Not eXists์˜ ์ค€๋ง์ด๋‹ค. ํ‚ค๊ฐ€ ์—†๋Š” ๋ฌธ์ž์—ด ๊ฐ’์„ ์œ ์ง€ํ•˜๋„๋ก ํ‚ค๋ฅผ ์„ค์ •ํ•œ๋‹ค. ์ด ๊ฒฝ์šฐ์—๋Š” SET๊ณผ ๋™์ผํ•˜๋‹ค. ์ฐจ์ด์ ์€ ํ‚ค๊ฐ€ ์ด๋ฏธ ๊ฐ’์„ ๋ณด์œ ํ•˜๊ณ  ์žˆ๋Š” ๊ฒฝ์šฐ ์ž‘์—…์ด ์ˆ˜ํ–‰๋˜์ง€ ์•Š๋Š”๋‹ค. ์ฆ‰ ์ด๊ฒƒ์„ ํ†ตํ•ด ๋ถ„์‚ฐ๋ฝ์„ ๊ตฌํ˜„ํ•  ์ˆ˜ ์žˆ๋‹ค.

# redis-cli
127.0.0.1:6379> SETNX 1 lock
(integer) 1 // ๋ฝ ํš๋“
127.0.0.1:6379> SETNX 1 lock
(integer) 0 // ๋ฝ ํš๋“ ์‹คํŒจ

SET์€ ๊ฐ€์žฅ ๊ธฐ๋ณธ์ ์ธ ๋ฐ์ดํ„ฐ ์ž…๋ ฅ ํ˜•ํƒœ๋กœ ํ‚ค์™€ ๊ฐ’์„ ์ž…๋ ฅํ•œ๋‹ค. ๋งŒ์•ฝ ํ‚ค๊ฐ€ ์ด๋ฏธ ์ง€์ •๋˜์–ด ์žˆ๋‹ค๋ฉด ๋ฎ์–ด์”Œ์›Œ ์ง„๋‹ค.

๋ ˆ๋””์Šค๋ฅผ ํ™œ์šฉ ํ–ˆ์„ ๋•Œ์˜ ๊ฐ€์žฅ ํฐ ์žฅ์ ์€ ๋””์Šคํฌ์— ์ ‘๊ทผํ•˜๋Š” ๊ฒƒ์ด ์•„๋‹ˆ๋ผ ๋ฉ”๋ชจ๋ฆฌ์— ์ ‘๊ทผํ•˜๋Š” ๊ฒƒ ์ด๊ธฐ ๋•Œ๋ฌธ์— ๋” ๋น ๋ฅด๊ฒŒ ๋ฝ์„ ํš๋“ํ•˜๊ณ  ํ•ด์ œํ•  ์ˆ˜ ์žˆ๋‹ค.

# Redis ์˜์กด์„ฑ

๋‹ค๋ฅธ ํ”„๋กœ๊ทธ๋ž˜๋ฐ ์–ธ์–ด์—์„œ ๋ ˆ๋””์Šค ํ”„๋กœํ† ์ฝœ ์‚ฌ์šฉ์„ ์œ„ํ•ด์„œ๋Š” ๋ ˆ๋””์Šค ํด๋ผ์ด์–ธํŠธ๋ฅผ ์‚ฌ์šฉํ•ด์•ผ ํ•œ๋‹ค. Java์—๋Š” ๋Œ€ํ‘œ์ ์œผ๋กœ Jedis, lettuce, Redisson ๋“ฑ์ด ์กด์žฌํ•œ๋‹ค.

img.png

๋จผ์ € ์Šคํ”„๋ง์—์„œ ๋ ˆ๋””์Šค ์‚ฌ์šฉ์„ ์œ„ํ•ด์„œ๋Š” ์˜์กด์„ฑ์„ ์ถ”๊ฐ€ํ•ด์•ผ ํ•œ๋‹ค.

dependencies {
    // ...
    implementation 'org.springframework.boot:spring-boot-starter-data-redis'
    // ...
}

๋Œ€๊ทœ๋ชจ Spring Data ์ œํ’ˆ๊ตฐ์˜ ์ผ๋ถ€์ธ Spring Data Redis๋Š” ์Šคํ”„๋ง ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜์—์„œ ์‰ฝ๊ฒŒ ๊ตฌ์„ฑํ•˜๊ณ  ๋ ˆ๋””์Šค์— ์ ‘๊ทผํ•  ์ˆ˜ ์žˆ๋„๋ก ์ง€์›ํ•œ๋‹ค. ์ด๊ฒƒ์€ ์ €์žฅ์†Œ์™€ ์ƒํ™” ์ž‘์šฉํ•˜๊ธฐ ์œ„ํ•ด ๋‚ฎ์€ ์ˆ˜์ค€๊ณผ ๋†’์€ ์ˆ˜์ค€์˜ ์ถ”์ƒํ™”๋ฅผ ๋ชจ๋‘ ์ œ๊ณตํ•˜๊ณ  ์žˆ๋‹ค. ์œ„ ์˜์กด์„ฑ์„ ์ถ”๊ฐ€ํ•˜๋ฉด ๋ ˆ๋””์Šค ํด๋ผ์ด์–ธํŠธ๊ฐ€ ๊ธฐ๋ณธ์ ์œผ๋กœ lettuce์— ๋Œ€ํ•œ ์˜์กด์„ฑ์ด ์ถ”๊ฐ€๋œ๋‹ค.

img.png

Lettuce๋Š” ๋™๊ธฐ์‹, ๋น„๋™๊ธฐ์‹ ๋ฐ reactive ์‚ฌ์šฉ์„ ์œ„ํ•œ ํ™•์žฅ ๊ฐ€๋Šฅํ•œ ์Šค๋ ˆ๋“œ ์•ˆ์ „ํ•œ ๋ ˆ๋””์Šค ํด๋ผ์ด์–ธํŠธ์ด๋‹ค.

# RedisTemplate

๋ ˆ๋””์Šค ๋ฐ์ดํ„ฐ ์ ‘๊ทผ ์ฝ”๋“œ๋ฅผ ๋‹จ์ˆœํ™”ํ•˜๋„๋ก ๋„์™€์ฃผ๋Š” ํด๋ž˜์Šค์ด๋‹ค. ๋ ˆ๋””์Šค ์ €์žฅ์†Œ์—์„œ ์ง€์ •๋œ ๊ฐ์ฒด์™€ ๊ธฐ๋ณธ ์ด์ง„ ๋ฐ์ดํ„ฐ ๊ฐ„์— ์ž๋™ ์ง๋ ฌํ™”/์—ญ์ง๋ ฌํ™”๋ฅผ ์ˆ˜ํ–‰ํ•œ๋‹ค. ๊ธฐ๋ณธ์ ์œผ๋กœ JdkSerializationRedisSerializer์„ ํ†ตํ•ด ๊ฐ์ฒด์— Java ์ง๋ ฌํ™”๋ฅผ ์‚ฌ์šฉํ•œ๋‹ค. ๋ฌธ์ž์—ด ์ง‘์ค‘ ์ž‘์—…์˜ ๊ฒฝ์šฐ ์ „์šฉ StringRedisTemplate๋ฅผ ๊ณ ๋ คํ•  ์ˆ˜ ์žˆ๋‹ค.

RedisTemplate์€ ๋ ˆ๋””์Šค๊ฐ€ ์ œ๊ณตํ•˜๋Š” ๋‹ค์–‘ํ•œ ๋ฐ์ดํ„ฐ ๊ตฌ์กฐ๋ฅผ ์ง€์›ํ•˜๊ธฐ ์œ„ํ•ด opsForXXX()์™€ ๊ฐ™์€ ํ˜•ํƒœ์˜ ๋ฉ”์„œ๋“œ๊ฐ€ ์กด์žฌํ•œ๋‹ค.

์‚ฌ์šฉํ•˜๊ณ ์ž ํ•˜๋Š” redis command์— ๋Œ€์‘ํ•˜๋Š” ๋ฉ”์„œ๋“œ๋ฅผ ํ˜ธ์ถœํ•˜๋ฉด operation ๊ฐ์ฒด๊ฐ€ ๋ฐ˜ํ™˜๋œ๋‹ค.

ValueOperations<String, String> valueOperations = redisTemplate.opsForValue();
  • opsForValue(): ValueOperations
  • opsForHash(): HashOperations
  • opsForList(): ListOperations
  • opsForSet(): SetOperations
  • opsForZSet(): ZSetOperations
  • opsForStream(): StreamOperations
  • opsForGeo(): GeoOperations
  • opsForHyperLogLog(): HyperLogLogOperations
  • opsForCluster(): ClusterOperations

๋ถ„์‚ฐ๋ฝ ๊ตฌํ˜„์„ ์œ„ํ•ด์„œ๋Š” ๊ฐ„๋‹จํžˆ ๋ฌธ์ž์—ด์„ ํ™œ์šฉํ•  ์ˆ˜ ์žˆ์„ ๊ฒƒ์ด๋‹ค. MySQL์˜ ๋„ค์ž„๋“œ ๋ฝ๊ณผ ๋™์ผํ•˜๊ฒŒ ํŠน์ • ๋ฌธ์ž์—ด์„ SETNX๋ฅผ ํ†ตํ•ด ๋ฝ์„ ๊ตฌํ˜„ํ•  ์ˆ˜ ์žˆ๋‹ค.

์•„๋ž˜๋Š” ์‹ค์ œ ๋ฝ์„ ํš๋“ํ•˜๊ณ  ํ•ด์ œํ•˜๊ธฐ ์œ„ํ•œ LockRepository์ด๋‹ค.


@Component
public class RedisLockRepository {

    private final StringRedisTemplate redisTemplate;

    public RedisLockRepository(final StringRedisTemplate stringRedisTemplate) {
        this.redisTemplate = stringRedisTemplate;
    }

    public Boolean lock(final Long key) {
        return redisTemplate.opsForValue()
                .setIfAbsent(generateKey(key), "lock", Duration.ofMillis(3_000));
    }

    public Boolean unlock(final Long key) {
        return redisTemplate.delete(generateKey(key));
    }

    private String generateKey(final Long key) {
        return key.toString();
    }
}
  • StringRedisTemplate: RedisTemplate์˜ ๋ฌธ์ž์—ด ์ค‘์‹ฌ์˜ ํด๋ž˜์Šค์ด๋‹ค. ๋ ˆ๋””์Šค์— ๋Œ€ํ•œ ๋Œ€๋ถ€๋ถ„์˜ ์ž‘์—…์€ ๋ฌธ์ž์—ด ๊ธฐ๋ฐ˜์ด๊ธฐ ๋•Œ๋ฌธ์— ์ด ํด๋ž˜์Šค๋Š” ํŠนํžˆ ์ง๋ ฌํ™” ๋ณ€ํ™˜๊ธฐ ์ธก๋ฉด์—์„œ ๋ณด๋‹ค ์ผ๋ฐ˜์ ์ธ ํ…œํ”Œ๋ฆฟ์˜ ๊ตฌ์„ฑ์„ ์ตœ์†Œํ™”ํ•˜๋Š” ์ „์šฉ ํด๋ž˜์Šค๋ฅผ ์ œ๊ณตํ•œ๋‹ค.
  • lock(): ๋ฝ ํš๋“์„ ์œ„ํ•œ ๋ฉ”์„œ๋“œ์ด๋‹ค. ๋ฝ ํš๋“์— ์„ฑ๊ณตํ•˜๋ฉด true, ์‹คํŒจํ•˜๋ฉด false๋ฅผ ๋ฐ˜ํ™˜ํ•œ๋‹ค.
  • opsForValue(): ๋‹จ์ˆœ ๊ฐ’์— ๋Œ€ํ•ด ์ˆ˜ํ–‰๋œ ์ž‘์—…์„ ๋ฐ˜ํ™˜ํ•œ๋‹ค.
  • setIfAbsent(): ํ‚ค๊ฐ€ ์—†๋Š” ๊ฒฝ์šฐ ๋ฌธ์ž์—ด value๋ฅผ ๋ณด์œ ํ•˜๋„๋ก ํ‚ค๋ฅผ ์„ค์ •ํ•œ๋‹ค. SETNX ๋ช…๋ น์–ด๋ฅผ ์ˆ˜ํ–‰ํ•˜๋Š” ๊ฒƒ์œผ๋กœ ํŒ๋‹จ๋œ๋‹ค. ํŠน๋ณ„ํ•œ ์กฐ์น˜๊ฐ€ ์—†๋‹ค๋ฉด ๋ ˆ๋””์Šค์˜ ํ‚ค๋Š” ์‚ญ์ œ๊ฐ€ ์•„๋‹Œ ๋ณด๊ด€ ์ฒ˜๋ฆฌ๋œ๋‹ค. ์ถ”์ธกํ•˜๊ธฐ๋กœ๋Š” timeout ์‹œ๊ฐ„์„ ์ง€์ •ํ•˜์—ฌ ์ž๋™์ ์œผ๋กœ ํ‚ค๋ฅผ ์ œ๊ฑฐํ•˜์—ฌ ๋ฝ์ด ์˜ค๋žœ์‹œ๊ฐ„ ์žกํžˆ๋Š” ๊ฒƒ์„ ๋ง‰๋Š” ๊ฒƒ์œผ๋กœ ์ถ”์ธก๋œ๋‹ค.
  • unlock(): ๋ฝ์„ ํ•ด์ œํ•œ๋‹ค.

๋ฝ์„ ํš๋“ํ•œ ๋’ค ๋กœ์ง์„ ์ˆ˜ํ–‰ํ•˜๊ณ  ๋ฝ์„ ํ•ด์ œํ•˜๊ธฐ ์œ„ํ•œ Facade๋ฅผ ์ถ”๊ฐ€ํ•œ๋‹ค.

@Component
public class LockProductFacade {

    private final RedisLockRepository redisLockRepository;
    private final ProductService productService;

    public LockProductFacade(final RedisLockRepository redisLockRepository,
                                 final ProductService productService) {
        this.redisLockRepository = redisLockRepository;
        this.productService = productService;
    }

    public void purchase(final Long key, final Long quantity) {
        try {
            redisLockRepository.lock(key);
            productService.purchase(key, quantity);
        } finally {
            redisLockRepository.unlock(key);
        }
    }
}

์ž ์ด์ œ ํ…Œ์ŠคํŠธ๋ฅผ ์ˆ˜ํ–‰ํ•ด๋ณด์ž.

@SpringBootTest
@DisplayNameGeneration(ReplaceUnderscores.class)
class SpinlockProductFacadeTest {

    private final ProductRepository productRepository;
    private final SpinlockProductFacade spinlockProductFacade;

    @Autowired
    SpinlockProductFacadeTest(final ProductRepository productRepository,
                              final SpinlockProductFacade spinlockProductFacade) {
        this.productRepository = productRepository;
        this.spinlockProductFacade = spinlockProductFacade;
    }

    @Test
    void ๋™์‹œ์—_100๊ฐœ์˜_์ƒํ’ˆ์„_๊ตฌ๋งคํ•œ๋‹ค() throws InterruptedException {
        var product = productRepository.save(new Product("์น˜ํ‚จ", 100L));

        var executorService = Executors.newFixedThreadPool(10);
        var countDownLatch = new CountDownLatch(100);
        for (int i = 0; i < 100; i++) {
            executorService.submit(() -> process(product, countDownLatch));
        }

        countDownLatch.await();

        var actual = productRepository.findById(product.getId()).orElseThrow();

        assertThat(actual.getQuantity()).isEqualTo(0L);
    }

    @Test
    void ์ƒํ’ˆ์˜_์ˆ˜๋Ÿ‰์€_100๊ฐœ_์ด์ง€๋งŒ_๋™์‹œ์—_200๊ฐœ์˜_์ƒํ’ˆ์„_๊ตฌ๋งคํ•œ๋‹ค() throws InterruptedException {
        var product = productRepository.save(new Product("์น˜ํ‚จ", 100L));

        var executorService = Executors.newFixedThreadPool(10);
        var countDownLatch = new CountDownLatch(200);
        for (int i = 0; i < 200; i++) {
            executorService.submit(() -> process(product, countDownLatch));
        }

        countDownLatch.await();

        var actual = productRepository.findById(product.getId()).orElseThrow();

        assertThat(actual.getQuantity()).isEqualTo(0L);
    }

    private void process(final Product product, final CountDownLatch countDownLatch) {
        try {
            spinlockProductFacade.purchase(product.getId(), 1L);
        } catch (final InterruptedException e) {
            throw new RuntimeException(e);
        } finally {
            countDownLatch.countDown();
        }
    }
}

img.png

์•„์‰ฝ๊ฒŒ๋„ ์‹คํŒจํ•œ๋‹ค. ์œ„ ๋ฐฉ์‹์„ ํ™œ์šฉํ•˜๋ฉด ๋ฝ ์‹คํŒจ ์‹œ ์ถ”๊ฐ€์ ์ธ ์žฌ์‹œ๋„ ๋กœ์ง์„ ์ž‘์„ฑํ•ด์•ผ ํ•œ๋‹ค. ๋ณดํ†ต์€ ํŠน์ • ์‹œ๊ฐ„ ์ดํ›„ ์žฌ์š”์ฒญ์„ ํ•  ์ˆ˜ ์žˆ๋Š” ์Šคํ•€๋ฝ ๋ฐฉ์‹์„ ํ™œ์šฉํ•  ์ˆ˜ ์žˆ๋‹ค.

# ์Šคํ•€๋ฝ ์ ์šฉํ•˜๊ธฐ

@Component
public class SpinlockProductFacade {

    private final RedisLockRepository redisLockRepository;
    private final ProductService productService;

    public SpinlockProductFacade(final RedisLockRepository redisLockRepository,
                                 final ProductService productService) {
        this.redisLockRepository = redisLockRepository;
        this.productService = productService;
    }

    public void purchase(final Long key, final Long quantity) throws InterruptedException {
        while (!redisLockRepository.lock(key)) {
            Thread.sleep(100);
        }

        try {
            productService.purchase(key, quantity);
        } finally {
            redisLockRepository.unlock(key);
        }
    }
}

๋กœ์ง์€ ๊ฐ„๋‹จํ•œ๋‹ค. ์Šคํ•€๋ฝ์„ ํ†ตํ•ด ๋ฝ์ด ํš๋“๊ฐ€๋Šฅํ•œ์ง€ ์ฃผ๊ธฐ์ ์œผ๋กœ ํ™•์ธ ํ›„ ๋กœ์ง์„ ์ˆ˜ํ–‰ํ•œ๋‹ค. ๋กœ์ง์ด ์ •์ƒ์ ์œผ๋กœ ์ˆ˜ํ–‰๋˜๋ฉด ๋ฝ์„ ํ•ด์ œํ•ด์•ผ ํ•˜๊ธฐ ๋•Œ๋ฌธ์— finally๋กœ ๋ฝ์„ ํ•ด์ œํ•œ๋‹ค.

ํ…Œ์ŠคํŠธ๋ฅผ ์ˆ˜ํ–‰ํ•˜๋ฉด ์ •์ƒ์ ์œผ๋กœ ํ†ต๊ณผํ•˜๋Š” ๊ฒƒ์„ ์•Œ ์ˆ˜ ์žˆ๋‹ค.

img.png

# ์ •๋ฆฌ

์ง€๊ธˆ ๊นŒ์ง€ ๋ ˆ๋””์Šค ํด๋ผ์ด์–ธํŠธ์ธ lettuce๋ฅผ ํ™œ์šฉํ•˜์—ฌ ๋™์‹œ์„ฑ ์ด์Šˆ๋ฅผ ํ•ด๊ฒฐํ•˜์˜€๋‹ค. lettuce๋Š” Spring Data Redis ์˜์กด์„ฑ ์ถ”๊ฐ€ ์‹œ ๊ธฐ๋ณธ์ ์œผ๋กœ ์ ์šฉ๋˜๊ธฐ ๋•Œ๋ฌธ์— ๊ฐ„ํŽธํ•˜๊ฒŒ ์ ์šฉ์ด ๊ฐ€๋Šฅํ•˜์ง€๋งŒ ๋ฝ ํš๋“ ๋ฐ ๋ฐ˜๋‚ฉ ๋กœ์ง์„ ๊ฐœ๋ฐœ์ž๊ฐ€ ์ง์ ‘ ์ž‘์„ฑํ•ด์•ผ ํ•œ๋‹ค.

๋˜ํ•œ ๋ฝ ํš๋“ ์‹คํŒจ ์‹œ ์žฌ์‹œ๋„๋ฅผ ์œ„ํ•œ ๋กœ์ง๊นŒ์ง€ ๊ตฌํ˜„ํ•ด์•ผ ํ•œ๋‹ค. ๋Œ€ํ‘œ์ ์œผ๋กœ๋Š” ์Šคํ•€๋ฝ์„ ํ™œ์šฉํ•˜์—ฌ ๊ตฌํ˜„ํ•  ์ˆ˜ ์žˆ๋‹ค. ์Šคํ•€๋ฝ ๋ฐฉ์‹์œผ๋กœ๋„ ์ถฉ๋ถ„ํžˆ ๋™์‹œ์„ฑ ์ด์Šˆ๊ฐ€ ํ•ด๊ฒฐ๋˜์ง€๋งŒ ๋ ˆ๋””์Šค ์„œ๋ฒ„์— ์ง€์†์ ์ธ ์š”์ฒญ์„ ๋ณด๋‚ด๊ธฐ ๋•Œ๋ฌธ์— ๋ถ€ํ•˜๊ฐ€ ์˜ฌ ์ˆ˜ ์žˆ๋‹ค.

๋‹ค์Œ ์‹œ๊ฐ„์—๋Š” pub/sub ๋ฐฉ์‹์„ ๊ธฐ๋ฐ˜์œผ๋กœ ๋ฝ์„ ๊ตฌํ˜„ํ•˜๋Š” ๋ ˆ๋””์Šค ํด๋ผ์ด์–ธํŠธ Redisson์„ ํ™œ์šฉํ•˜์—ฌ ๋™์‹œ์„ฑ ์ด์Šˆ๋ฅผ ํ•ด๊ฒฐํ•ด๋ณด๋ ค ํ•œ๋‹ค.

# References.

SETNX (opens new window)
setIfAbsent() (opens new window)
Redis client (opens new window)
Lettuce - Advanced Java Redis client (opens new window)
Spring Boot Data Redis ์‚ฌ์šฉํ•ด๋ณด๊ธฐ (opens new window)

last updated: 12/16/2022, 11:49:18 PM